[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:02 dist-test-slave-0b58 krb5kdc[16475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:02 dist-test-slave-0b58 krb5kdc[16475](info): set up 2 sockets
Aug 06 10:47:02 dist-test-slave-0b58 krb5kdc[16475](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 10:47:03.111789 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:03 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:03.312235 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:37513
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:03.641405 16488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:03.641685 16488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:03.641870 16488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:03.649184 16488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:03.649296 16488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:03.649411 16488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:03.649516 16488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:03.649605 16488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:03.659602 16488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:03.660955 16488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:03.662263 16488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:03.674867 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:03.674924 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:03.674837 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:03.675429 16488 server_base.cc:1034] running on GCE node
I20260806 10:47:03.676484 16488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:03.677942 16488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:03.679191 16488 hybrid_clock.cc:648] HybridClock initialized: now 1786013223679137 us; error 74 us; skew 500 ppm
I20260806 10:47:03.679541 16488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:03 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:03.682797 16488 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:03.684423 16488 webserver.cc:460] Webserver started at http://127.16.21.62:42001/ using document root <none> and password file <none>
I20260806 10:47:03.685148 16488 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:03.685305 16488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:03.685618 16488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:03.688884 16488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "83ff495ba84d4fa2bdc2cbc965c02123"
format_stamp: "Formatted at 2026-08-06 10:47:03 on dist-test-slave-0b58"
server_key: "bfa1674690fb83e14d8d469cfc287abe"
server_key_iv: "6abff8e043454a2ced96b2ccdbb2d9e8"
server_key_version: "encryptionkey@0"
I20260806 10:47:03.689739 16488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "83ff495ba84d4fa2bdc2cbc965c02123"
format_stamp: "Formatted at 2026-08-06 10:47:03 on dist-test-slave-0b58"
server_key: "bfa1674690fb83e14d8d469cfc287abe"
server_key_iv: "6abff8e043454a2ced96b2ccdbb2d9e8"
server_key_version: "encryptionkey@0"
I20260806 10:47:03.694914 16488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:47:03.698411 16503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:03.699678 16488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:03.699868 16488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "83ff495ba84d4fa2bdc2cbc965c02123"
format_stamp: "Formatted at 2026-08-06 10:47:03 on dist-test-slave-0b58"
server_key: "bfa1674690fb83e14d8d469cfc287abe"
server_key_iv: "6abff8e043454a2ced96b2ccdbb2d9e8"
server_key_version: "encryptionkey@0"
I20260806 10:47:03.700071 16488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:03.719456 16488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:03.723542 16488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:03.723845 16488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:03.743120 16488 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:37513
I20260806 10:47:03.743160 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:37513 every 8 connection(s)
I20260806 10:47:03.744462 16488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:03.748039 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:03.750694 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 16488
I20260806 10:47:03.751037 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:03.752986 16468 external_mini_cluster.cc:1428] Setting key 958b4d6cbad1a9cb67a76cb6d6025094
I20260806 10:47:03.758574 16556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123: Bootstrap starting.
I20260806 10:47:03.761602 16556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:03.762624 16556 log.cc:824] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:03.765318 16556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123: No bootstrap required, opened a new log
I20260806 10:47:03.775645 16556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 37513 } }
I20260806 10:47:03.776113 16556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:03.776222 16556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ff495ba84d4fa2bdc2cbc965c02123, State: Initialized, Role: FOLLOWER
I20260806 10:47:03.776916 16556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 37513 } }
I20260806 10:47:03.777201 16556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:03.777377 16556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:03.777563 16556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:03.779923 16556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 37513 } }
I20260806 10:47:03.780483 16556 leader_election.cc:302] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ff495ba84d4fa2bdc2cbc965c02123; no voters:
I20260806 10:47:03.781119 16556 leader_election.cc:288] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:03.781236 16561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 1 FOLLOWER]: Leader election won for term 1
Aug 06 10:47:03 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:03.782070 16561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [term 1 LEADER]: Becoming Leader. State: Replica: 83ff495ba84d4fa2bdc2cbc965c02123, State: Running, Role: LEADER
I20260806 10:47:03.782641 16561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 37513 } }
I20260806 10:47:03.783179 16556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:03.786100 16563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ff495ba84d4fa2bdc2cbc965c02123. Latest consensus state: current_term: 1 leader_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 37513 } } }
I20260806 10:47:03.786562 16563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:03.786940 16562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff495ba84d4fa2bdc2cbc965c02123" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 37513 } } }
I20260806 10:47:03.787228 16562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:03.798135 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:03.756738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.16.21.62:37513)
0806 10:47:03.757681 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:03.757695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:03.759098 (+ 1403us) server_negotiation.cc:407] Connection header received
0806 10:47:03.768082 (+ 8984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:03.768106 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:03.768623 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:03.769038 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:03.771723 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:03.773823 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:03.775456 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:03.776541 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:03.776726 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:03.783512 (+ 6786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:03.783524 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:03.783541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:03.783589 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:03.788580 (+ 4991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:03.789892 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:03.789903 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:03.789914 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:03.790165 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:03.790873 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:03.790881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:03.790891 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:03.791776 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:03.795546 (+ 3770us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:03.795892 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 10:47:03.796440 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":422,"threads_started":1}
I20260806 10:47:03.802845 16566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:03.808089 16566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:03.819826 16566 catalog_manager.cc:1347] Generated new cluster ID: c8e91a560d82428eaf2bdff9cdef4850
I20260806 10:47:03.819977 16566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:03.829420 16566 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:03.830689 16566 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:03.841900 16566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83ff495ba84d4fa2bdc2cbc965c02123: Generated new TSK 0
I20260806 10:47:03.842690 16566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:03.904042 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:37513
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:04.246757 16582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:04.247063 16582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:04.247239 16582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:04.254912 16582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:04.255059 16582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:04.255249 16582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:04.265056 16582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:37513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:04.266330 16582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:04.267644 16582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:04.281708 16587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:04.281625 16590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:04.281654 16588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:04.287533 16582 server_base.cc:1034] running on GCE node
I20260806 10:47:04.288151 16582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:04.289242 16582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:04.290483 16582 hybrid_clock.cc:648] HybridClock initialized: now 1786013224290445 us; error 58 us; skew 500 ppm
I20260806 10:47:04.290804 16582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:04 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:04.294090 16582 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:04.295547 16582 webserver.cc:460] Webserver started at http://127.16.21.1:43921/ using document root <none> and password file <none>
I20260806 10:47:04.296250 16582 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:04.296464 16582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:04.296794 16582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:04.299612 16582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "06ebd18025954397aec1a6e2542b7c08"
format_stamp: "Formatted at 2026-08-06 10:47:04 on dist-test-slave-0b58"
server_key: "c6a5ae8d668fe413a95f194b38086b77"
server_key_iv: "4178a3ace2b14027d023e4e0e85c1611"
server_key_version: "encryptionkey@0"
I20260806 10:47:04.300450 16582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "06ebd18025954397aec1a6e2542b7c08"
format_stamp: "Formatted at 2026-08-06 10:47:04 on dist-test-slave-0b58"
server_key: "c6a5ae8d668fe413a95f194b38086b77"
server_key_iv: "4178a3ace2b14027d023e4e0e85c1611"
server_key_version: "encryptionkey@0"
I20260806 10:47:04.305507 16582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:04.308833 16597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:04.310127 16582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:04.310305 16582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "06ebd18025954397aec1a6e2542b7c08"
format_stamp: "Formatted at 2026-08-06 10:47:04 on dist-test-slave-0b58"
server_key: "c6a5ae8d668fe413a95f194b38086b77"
server_key_iv: "4178a3ace2b14027d023e4e0e85c1611"
server_key_version: "encryptionkey@0"
I20260806 10:47:04.310504 16582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:04.324788 16582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:04.328531 16582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:04.328814 16582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:04.329726 16582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:04.330994 16582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:04.331106 16582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:04.331261 16582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:04.331350 16582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:04.366617 16582 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:44389
I20260806 10:47:04.366659 16713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:44389 every 8 connection(s)
I20260806 10:47:04.367939 16582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:04.375065 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 16582
I20260806 10:47:04.375322 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:04.376153 16468 external_mini_cluster.cc:1428] Setting key ec8f84a74ca5ce39837533611222415d
Aug 06 10:47:04 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:04.390671 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:04.370617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:42961 (local address 127.16.21.62:37513)
0806 10:47:04.371113 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:04.371118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:04.371928 (+ 810us) server_negotiation.cc:407] Connection header received
0806 10:47:04.373903 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:04.373909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:04.374004 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:04.374135 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:04.376437 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.377412 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.379682 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.380084 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.380186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:04.384660 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:04.384666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:04.384669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:04.384701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:04.387492 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:04.388233 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:04.388240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:04.388243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:04.388403 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:04.388917 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:04.388924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:04.388927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:04.389190 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:04.389354 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:04.390066 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 10:47:04.390301 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":271,"threads_started":1}
I20260806 10:47:04.391641 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:04.371046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:37513 (local address 127.16.21.1:42961)
0806 10:47:04.371762 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:04.371803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:04.373616 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:04.374386 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:04.374395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:04.375193 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:04.376057 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:04.376083 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.377581 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.377585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:04.379519 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:04.379537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.380367 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.380378 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:04.380725 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:04.381479 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:04.381516 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:04.384417 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:04.387648 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:04.387657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:04.387672 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:04.388103 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:04.388523 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:04.388529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:04.388533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:04.388793 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:04.389335 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:04.389343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:04.389670 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 10:47:04.389936 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 10:47:04.390462 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":299,"threads_started":1}
I20260806 10:47:04.393447 16714 heartbeater.cc:347] Connected to a master server at 127.16.21.62:37513
I20260806 10:47:04.393757 16714 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:04.394500 16714 heartbeater.cc:511] Master 127.16.21.62:37513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
I20260806 10:47:04.396910 16521 ts_manager.cc:194] Registered new tserver with Master: 06ebd18025954397aec1a6e2542b7c08 (127.16.21.1:44389)
I20260806 10:47:04.398578 16521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:42961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:04.428363 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:37513
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:04.757670 16720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:04.757931 16720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:04.758119 16720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:04.765376 16720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:04.765511 16720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:04.765710 16720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:04.775204 16720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:37513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:04.776541 16720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:04.777870 16720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:04.791250 16725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:04.791326 16726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:04.791383 16728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:04.792048 16720 server_base.cc:1034] running on GCE node
I20260806 10:47:04.792753 16720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:04.793673 16720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:04.794874 16720 hybrid_clock.cc:648] HybridClock initialized: now 1786013224794817 us; error 49 us; skew 500 ppm
I20260806 10:47:04.795194 16720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:04 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:04.798270 16720 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:04.799808 16720 webserver.cc:460] Webserver started at http://127.16.21.2:46663/ using document root <none> and password file <none>
I20260806 10:47:04.800694 16720 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:04.800918 16720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:04.801358 16720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:04.804386 16720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "13c741348e8a479cae367c472430115b"
format_stamp: "Formatted at 2026-08-06 10:47:04 on dist-test-slave-0b58"
server_key: "77c3c7ca77f974426be6078205824788"
server_key_iv: "98dcb322c4689c6ba1d4517884ca3d6e"
server_key_version: "encryptionkey@0"
I20260806 10:47:04.805092 16720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "13c741348e8a479cae367c472430115b"
format_stamp: "Formatted at 2026-08-06 10:47:04 on dist-test-slave-0b58"
server_key: "77c3c7ca77f974426be6078205824788"
server_key_iv: "98dcb322c4689c6ba1d4517884ca3d6e"
server_key_version: "encryptionkey@0"
I20260806 10:47:04.810348 16720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260806 10:47:04.813902 16735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:04.815271 16720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:47:04.815467 16720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "13c741348e8a479cae367c472430115b"
format_stamp: "Formatted at 2026-08-06 10:47:04 on dist-test-slave-0b58"
server_key: "77c3c7ca77f974426be6078205824788"
server_key_iv: "98dcb322c4689c6ba1d4517884ca3d6e"
server_key_version: "encryptionkey@0"
I20260806 10:47:04.815685 16720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:04.846573 16720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:04.850535 16720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:04.850849 16720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:04.851842 16720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:04.853209 16720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:04.853331 16720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:04.853498 16720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:04.853590 16720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:04.888475 16720 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:36285
I20260806 10:47:04.888521 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:36285 every 8 connection(s)
I20260806 10:47:04.889881 16720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:04.898093 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 16720
I20260806 10:47:04.898307 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:04.898784 16468 external_mini_cluster.cc:1428] Setting key 5de9ede05dd35e6841cc2da82fa86da2
Aug 06 10:47:04 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:04.914846 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:04.892471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:51381 (local address 127.16.21.62:37513)
0806 10:47:04.892911 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:04.892914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:04.893790 (+ 876us) server_negotiation.cc:407] Connection header received
0806 10:47:04.895590 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:04.895596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:04.895683 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:04.895813 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:04.897359 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.898405 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.900229 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.900820 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.900942 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:04.908966 (+ 8024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:04.908971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:04.908975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:04.909005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:04.911712 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:04.912505 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:04.912510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:04.912514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:04.912645 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:04.913084 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:04.913090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:04.913098 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:04.913365 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:04.913542 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:04.914247 (+ 705us) server_negotiation.cc:299] Negotiation successful
0806 10:47:04.914484 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260806 10:47:04.915685 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:04.892917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:37513 (local address 127.16.21.2:51381)
0806 10:47:04.893618 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:04.893654 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:04.895327 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:04.896040 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:04.896047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:04.896639 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:04.897157 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:04.897173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.898589 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.898594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:04.900091 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:04.900104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:04.901059 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:04.901064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:04.901245 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:04.906341 (+ 5096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:04.906363 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:04.908726 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:04.911854 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:04.911862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:04.911876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:04.912380 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:04.912778 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:04.912783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:04.912787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:04.912961 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:04.913542 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:04.913551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:04.913866 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 10:47:04.914125 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 10:47:04.914598 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":308,"threads_started":1}
I20260806 10:47:04.917148 16851 heartbeater.cc:347] Connected to a master server at 127.16.21.62:37513
I20260806 10:47:04.917454 16851 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:04.918136 16851 heartbeater.cc:511] Master 127.16.21.62:37513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
I20260806 10:47:04.919651 16521 ts_manager.cc:194] Registered new tserver with Master: 13c741348e8a479cae367c472430115b (127.16.21.2:36285)
I20260806 10:47:04.920593 16521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:51381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:04.947216 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:37513
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:05.290135 16857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:05.290393 16857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:05.290550 16857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:05.297714 16857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:05.297813 16857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:05.297984 16857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:05.307425 16857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:37513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:05.308619 16857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:05.309952 16857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:05.323210 16862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:05.323119 16863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:05.323173 16865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:05.324347 16857 server_base.cc:1034] running on GCE node
I20260806 10:47:05.324942 16857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:05.325763 16857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:05.326959 16857 hybrid_clock.cc:648] HybridClock initialized: now 1786013225326922 us; error 29 us; skew 500 ppm
I20260806 10:47:05.327306 16857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:05.330495 16857 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:05.331899 16857 webserver.cc:460] Webserver started at http://127.16.21.3:38899/ using document root <none> and password file <none>
I20260806 10:47:05.332604 16857 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:05.332716 16857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:05.332966 16857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:05.335712 16857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "5ffc7069c1714a9f97bc50bc7196605b"
format_stamp: "Formatted at 2026-08-06 10:47:05 on dist-test-slave-0b58"
server_key: "0aa611982bf205a22ea719a39bb08884"
server_key_iv: "f6545ab09f8d105066acbf25111686c4"
server_key_version: "encryptionkey@0"
I20260806 10:47:05.336525 16857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "5ffc7069c1714a9f97bc50bc7196605b"
format_stamp: "Formatted at 2026-08-06 10:47:05 on dist-test-slave-0b58"
server_key: "0aa611982bf205a22ea719a39bb08884"
server_key_iv: "f6545ab09f8d105066acbf25111686c4"
server_key_version: "encryptionkey@0"
I20260806 10:47:05.341565 16857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:47:05.344889 16872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:05.346181 16857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:05.346378 16857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "5ffc7069c1714a9f97bc50bc7196605b"
format_stamp: "Formatted at 2026-08-06 10:47:05 on dist-test-slave-0b58"
server_key: "0aa611982bf205a22ea719a39bb08884"
server_key_iv: "f6545ab09f8d105066acbf25111686c4"
server_key_version: "encryptionkey@0"
I20260806 10:47:05.346593 16857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:05.359498 16857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:05.363274 16857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:05.363572 16857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:05.364573 16857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:05.365818 16857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:05.365931 16857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:05.366191 16857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:05.366304 16857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:05.401886 16857 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:43391
I20260806 10:47:05.401932 16986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:43391 every 8 connection(s)
I20260806 10:47:05.402511 16714 heartbeater.cc:503] Master 127.16.21.62:37513 was elected leader, sending a full tablet report...
I20260806 10:47:05.403366 16857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:05.405053 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 16857
I20260806 10:47:05.405275 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:05.405813 16468 external_mini_cluster.cc:1428] Setting key 208c3bb201d82f88048d3389b19aa2ae
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:05.425055 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.406456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:40347 (local address 127.16.21.62:37513)
0806 10:47:05.406665 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:05.406671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:05.408092 (+ 1421us) server_negotiation.cc:407] Connection header received
0806 10:47:05.410346 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:05.410351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:05.410452 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:05.410574 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:05.412029 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.412891 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.414614 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.415039 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.415179 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.418485 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:05.418491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:05.418500 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:05.418532 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:05.421904 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.422697 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.422703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.422706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.422844 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.423295 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.423301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.423304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.423597 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:05.423768 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:05.424498 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 10:47:05.424728 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:47:05.426116 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.406982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:37513 (local address 127.16.21.3:40347)
0806 10:47:05.407939 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:05.407975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:05.410084 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:05.410802 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:05.410810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:05.411309 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:05.411859 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:05.411875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.413049 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.413062 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:05.414474 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:05.414486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.415274 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.415281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:05.415502 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.416113 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:05.416134 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:05.418279 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:05.422058 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:05.422066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:05.422082 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:05.422534 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:05.422965 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:05.422971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:05.422974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:05.423167 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:05.423743 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:05.423750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:05.424077 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 10:47:05.424393 (+ 316us) client_negotiation.cc:240] Negotiation successful
0806 10:47:05.424911 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":510,"threads_started":1}
I20260806 10:47:05.427358 16987 heartbeater.cc:347] Connected to a master server at 127.16.21.62:37513
I20260806 10:47:05.427673 16987 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:05.428467 16987 heartbeater.cc:511] Master 127.16.21.62:37513 requested a full tablet report, sending...
I20260806 10:47:05.429852 16521 ts_manager.cc:194] Registered new tserver with Master: 5ffc7069c1714a9f97bc50bc7196605b (127.16.21.3:43391)
I20260806 10:47:05.430650 16521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:40347
I20260806 10:47:05.434242 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:05.483604 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.468747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.16.21.1:44389)
0806 10:47:05.469416 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:05.469424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:05.469451 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:47:05.469512 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:05.469518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:05.469716 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:05.469835 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:05.471210 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.472237 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.473482 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.474031 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.474124 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.478242 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:05.478253 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:05.478273 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:05.478308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:05.481115 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.481697 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.481703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.481705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.481815 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.482159 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.482162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.482164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.482478 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:05.482720 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:05.483000 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 10:47:05.483249 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":327,"threads_started":1}
W20260806 10:47:05.484859 16627 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38614
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:05.530280 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.517038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.16.21.1:44389)
0806 10:47:05.517239 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:05.517245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:05.517450 (+ 205us) server_negotiation.cc:407] Connection header received
0806 10:47:05.517618 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:05.517621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:05.517692 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:05.517763 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:05.519136 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.519850 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.521153 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.521570 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.521664 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.525429 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:05.525459 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:05.525462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:05.525493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:05.527945 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.528564 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.528567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.528569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.528685 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.529066 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.529071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.529073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.529343 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:05.529526 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:05.529764 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 10:47:05.529918 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 10:47:05.531852 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 16582
I20260806 10:47:05.543668 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.532803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.16.21.1:44389)
0806 10:47:05.532981 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:05.532985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:05.533338 (+ 353us) server_negotiation.cc:407] Connection header received
0806 10:47:05.533441 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:05.533445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:05.533495 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:05.533570 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:05.534683 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.535351 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.536607 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.536992 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.537079 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.539265 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:05.539269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:05.539271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:05.539294 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:05.541541 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.542189 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.542193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.542195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.542325 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.542632 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.542635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.542637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.542852 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:05.543020 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:05.543272 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 10:47:05.543405 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:47:05.544428 16709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:05.627848 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 16582
I20260806 10:47:05.647883 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 16720
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:05.663982 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.648996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.16.21.2:36285)
0806 10:47:05.649652 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:05.649659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:05.649685 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:47:05.649754 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:05.649770 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:05.649970 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:05.650193 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:05.651438 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.652519 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.653796 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.654327 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.654476 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.658636 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:05.658647 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:05.658660 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:05.658690 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:05.661567 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.662146 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.662153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.662155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.662282 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.662605 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.662608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.662611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.662930 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:05.663137 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:05.663473 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 10:47:05.663660 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":339,"threads_started":1}
I20260806 10:47:05.665287 16846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:05.748785 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 16720
I20260806 10:47:05.768311 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 16857
Aug 06 10:47:05 dist-test-slave-0b58 krb5kdc[16475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:05.783792 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:05.769356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.16.21.3:43391)
0806 10:47:05.769965 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:05.769971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:05.769999 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:47:05.770056 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:05.770062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:05.770261 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:05.770432 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:05.771640 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.772701 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.774017 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:05.774561 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:05.774709 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:05.778736 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:05.778753 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:05.778765 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:05.778794 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:05.781362 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.781993 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.781999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.782001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.782111 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:05.782419 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:05.782422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:05.782424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:05.782719 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:05.783006 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:05.783311 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 10:47:05.783496 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":302,"threads_started":1}
I20260806 10:47:05.785228 16982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:05.867677 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 16857
I20260806 10:47:05.886988 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 16488
I20260806 10:47:05.888249 16551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:05.994449 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 16488
2026-08-06T10:47:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4035 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:06 dist-test-slave-0b58 krb5kdc[17008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:06 dist-test-slave-0b58 krb5kdc[17008](info): set up 2 sockets
Aug 06 10:47:06 dist-test-slave-0b58 krb5kdc[17008](info): commencing operation
krb5kdc: starting...
W20260806 10:47:07.095952 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:07 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:07.298025 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46565
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:46565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:07.625790 17021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:07.626075 17021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:07.626191 17021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:07.633457 17021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:07.633559 17021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:07.633617 17021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:07.633677 17021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:07.633728 17021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:07.644124 17021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:46565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:07.645463 17021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:07.646778 17021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:07.658959 17026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:07.658865 17029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:07.658902 17027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:07.659912 17021 server_base.cc:1034] running on GCE node
I20260806 10:47:07.660749 17021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:07.662115 17021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:07.663388 17021 hybrid_clock.cc:648] HybridClock initialized: now 1786013227663341 us; error 56 us; skew 500 ppm
I20260806 10:47:07.663789 17021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:07 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:07.677021 17021 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:07.678707 17021 webserver.cc:460] Webserver started at http://127.16.21.62:33223/ using document root <none> and password file <none>
I20260806 10:47:07.679450 17021 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:07.679618 17021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:07.679937 17021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:07.683014 17021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "efe6a19e71ba4ee5a7202f94f186daec"
format_stamp: "Formatted at 2026-08-06 10:47:07 on dist-test-slave-0b58"
server_key: "75e2e3b4481a02eafee4a450016d9e58"
server_key_iv: "aca7971b9c38beb3fd2b4d6ee833199c"
server_key_version: "encryptionkey@0"
I20260806 10:47:07.683815 17021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "efe6a19e71ba4ee5a7202f94f186daec"
format_stamp: "Formatted at 2026-08-06 10:47:07 on dist-test-slave-0b58"
server_key: "75e2e3b4481a02eafee4a450016d9e58"
server_key_iv: "aca7971b9c38beb3fd2b4d6ee833199c"
server_key_version: "encryptionkey@0"
I20260806 10:47:07.689065 17021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 10:47:07.692520 17037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:07.693742 17021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:47:07.693930 17021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "efe6a19e71ba4ee5a7202f94f186daec"
format_stamp: "Formatted at 2026-08-06 10:47:07 on dist-test-slave-0b58"
server_key: "75e2e3b4481a02eafee4a450016d9e58"
server_key_iv: "aca7971b9c38beb3fd2b4d6ee833199c"
server_key_version: "encryptionkey@0"
I20260806 10:47:07.694140 17021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:07.709666 17021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:07.713892 17021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:07.714179 17021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:07.733284 17021 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:46565
I20260806 10:47:07.733362 17088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:46565 every 8 connection(s)
I20260806 10:47:07.734589 17021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:07.735996 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17021
I20260806 10:47:07.736253 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:07.737160 16468 external_mini_cluster.cc:1428] Setting key 5fc8c99e623028c0d4ce8e7a2b47b472
I20260806 10:47:07.739131 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:07.750135 17089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec: Bootstrap starting.
Aug 06 10:47:07 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:07.753685 17089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:07.754741 17089 log.cc:824] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:07.757942 17089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec: No bootstrap required, opened a new log
I20260806 10:47:07.759255 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:07.739249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.16.21.62:46565)
0806 10:47:07.740003 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:07.740013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:07.740050 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:47:07.741872 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:07.741893 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:07.742243 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:07.742624 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:07.744120 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:07.745472 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:07.746702 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:07.747382 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:07.747584 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:07.751976 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:07.751983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:07.751995 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:07.752032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:07.755233 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:07.755923 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:07.755943 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:07.755951 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:07.756135 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:07.756596 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:07.756600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:07.756601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:07.757095 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:07.757341 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:07.757900 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 10:47:07.758317 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260806 10:47:07.765390 17089 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe6a19e71ba4ee5a7202f94f186daec" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46565 } }
I20260806 10:47:07.765801 17089 raft_consensus.cc:374] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:07.765919 17089 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe6a19e71ba4ee5a7202f94f186daec, State: Initialized, Role: FOLLOWER
I20260806 10:47:07.766460 17089 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe6a19e71ba4ee5a7202f94f186daec" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46565 } }
I20260806 10:47:07.766691 17089 raft_consensus.cc:388] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:07.766862 17089 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:07.767040 17089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:07.769270 17089 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe6a19e71ba4ee5a7202f94f186daec" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46565 } }
I20260806 10:47:07.769757 17089 leader_election.cc:302] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efe6a19e71ba4ee5a7202f94f186daec; no voters:
I20260806 10:47:07.770396 17089 leader_election.cc:288] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:07.770563 17094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:07.771270 17094 raft_consensus.cc:686] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [term 1 LEADER]: Becoming Leader. State: Replica: efe6a19e71ba4ee5a7202f94f186daec, State: Running, Role: LEADER
I20260806 10:47:07.771818 17094 consensus_queue.cc:237] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe6a19e71ba4ee5a7202f94f186daec" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46565 } }
I20260806 10:47:07.772538 17089 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:07.774737 17095 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efe6a19e71ba4ee5a7202f94f186daec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe6a19e71ba4ee5a7202f94f186daec" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46565 } } }
I20260806 10:47:07.775213 17095 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:07.776784 17096 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe6a19e71ba4ee5a7202f94f186daec. Latest consensus state: current_term: 1 leader_uuid: "efe6a19e71ba4ee5a7202f94f186daec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe6a19e71ba4ee5a7202f94f186daec" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46565 } } }
I20260806 10:47:07.777050 17101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:07.777092 17096 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:07.781386 17101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:07.789578 17101 catalog_manager.cc:1347] Generated new cluster ID: 619e5ff0c6df4164a45bd969f3df1d20
I20260806 10:47:07.789678 17101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:07.805169 17101 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:07.806351 17101 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:07.816798 17101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efe6a19e71ba4ee5a7202f94f186daec: Generated new TSK 0
I20260806 10:47:07.817502 17101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:07.870262 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46565
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:08.200440 17115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:08.200712 17115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:08.200899 17115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:08.208035 17115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:08.208141 17115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:08.208386 17115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:08.218029 17115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:08.219254 17115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:08.220616 17115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:08.235646 17120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:08.235719 17123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:08.235699 17121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:08.236670 17115 server_base.cc:1034] running on GCE node
I20260806 10:47:08.237268 17115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:08.238164 17115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:08.239362 17115 hybrid_clock.cc:648] HybridClock initialized: now 1786013228239323 us; error 29 us; skew 500 ppm
I20260806 10:47:08.239673 17115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:08 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:08.242871 17115 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:08.244338 17115 webserver.cc:460] Webserver started at http://127.16.21.1:42705/ using document root <none> and password file <none>
I20260806 10:47:08.244998 17115 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:08.245131 17115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:08.245446 17115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:08.248217 17115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "6ac0fc1e9f574d1e9d656c788538cbdc"
format_stamp: "Formatted at 2026-08-06 10:47:08 on dist-test-slave-0b58"
server_key: "d27bce06ff01e55179399f48cd0c911e"
server_key_iv: "0c59af56003f906db0e0ab0c814a67dc"
server_key_version: "encryptionkey@0"
I20260806 10:47:08.249092 17115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "6ac0fc1e9f574d1e9d656c788538cbdc"
format_stamp: "Formatted at 2026-08-06 10:47:08 on dist-test-slave-0b58"
server_key: "d27bce06ff01e55179399f48cd0c911e"
server_key_iv: "0c59af56003f906db0e0ab0c814a67dc"
server_key_version: "encryptionkey@0"
I20260806 10:47:08.253931 17115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:47:08.257252 17130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:08.258589 17115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:47:08.258777 17115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "6ac0fc1e9f574d1e9d656c788538cbdc"
format_stamp: "Formatted at 2026-08-06 10:47:08 on dist-test-slave-0b58"
server_key: "d27bce06ff01e55179399f48cd0c911e"
server_key_iv: "0c59af56003f906db0e0ab0c814a67dc"
server_key_version: "encryptionkey@0"
I20260806 10:47:08.258985 17115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:08.300482 17115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:08.304836 17115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:08.305153 17115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:08.306144 17115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:08.307468 17115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:08.307577 17115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:08.307734 17115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:08.307824 17115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:08.341332 17115 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:41303
I20260806 10:47:08.341408 17243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:41303 every 8 connection(s)
I20260806 10:47:08.342607 17115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:08.349295 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17115
I20260806 10:47:08.349512 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:08.350109 16468 external_mini_cluster.cc:1428] Setting key f851e42cd52bcf7b5313b562e726bb34
Aug 06 10:47:08 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:08.363445 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:08.346365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:43629 (local address 127.16.21.62:46565)
0806 10:47:08.346851 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:08.346857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:08.346879 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:47:08.348692 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:08.348698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:08.348789 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:08.348927 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:08.350546 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.351428 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.353048 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.353449 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.353589 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:08.357021 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:08.357026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:08.357029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:08.357058 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:08.360076 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:08.360876 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:08.360885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:08.360891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:08.361041 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:08.361465 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:08.361473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:08.361479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:08.361857 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:08.362016 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:08.362711 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 10:47:08.362955 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260806 10:47:08.364202 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:08.345833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46565 (local address 127.16.21.1:43629)
0806 10:47:08.346556 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:08.346610 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:08.348430 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:08.349159 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:08.349167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:08.349783 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:08.350376 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:08.350395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.351624 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.351628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:08.352892 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:08.352906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.353683 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.353691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:08.353925 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:08.354603 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:08.354624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:08.356805 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:08.360294 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:08.360304 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:08.360318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:08.360740 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:08.361161 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:08.361165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:08.361167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:08.361348 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:08.361979 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:08.362004 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:08.362305 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 10:47:08.362554 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 10:47:08.362897 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":306,"threads_started":1}
I20260806 10:47:08.365763 17244 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46565
I20260806 10:47:08.366053 17244 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:08.366778 17244 heartbeater.cc:511] Master 127.16.21.62:46565 requested a full tablet report, sending...
I20260806 10:47:08.368971 17054 ts_manager.cc:194] Registered new tserver with Master: 6ac0fc1e9f574d1e9d656c788538cbdc (127.16.21.1:41303)
I20260806 10:47:08.370395 17054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:43629
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:08.397359 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46565
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:08.722637 17251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:08.722901 17251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:08.723071 17251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:08.730369 17251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:08.730465 17251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:08.730685 17251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:08.744828 17251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:08.746187 17251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:08.747833 17251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:08.762574 17257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:08.762601 17259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:08.763319 17256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:08.763846 17251 server_base.cc:1034] running on GCE node
I20260806 10:47:08.764511 17251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:08.765409 17251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:08.766717 17251 hybrid_clock.cc:648] HybridClock initialized: now 1786013228766653 us; error 48 us; skew 500 ppm
I20260806 10:47:08.767060 17251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:08 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:08.770275 17251 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:08.771663 17251 webserver.cc:460] Webserver started at http://127.16.21.2:34373/ using document root <none> and password file <none>
I20260806 10:47:08.772413 17251 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:08.772578 17251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:08.772921 17251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:08.775806 17251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "5a4c639026de458c812a7a79e70e3413"
format_stamp: "Formatted at 2026-08-06 10:47:08 on dist-test-slave-0b58"
server_key: "d2756605e59c3deca6d94a723a28a8a4"
server_key_iv: "a837035b74db3c9f4c86e4d7635725ab"
server_key_version: "encryptionkey@0"
I20260806 10:47:08.776614 17251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "5a4c639026de458c812a7a79e70e3413"
format_stamp: "Formatted at 2026-08-06 10:47:08 on dist-test-slave-0b58"
server_key: "d2756605e59c3deca6d94a723a28a8a4"
server_key_iv: "a837035b74db3c9f4c86e4d7635725ab"
server_key_version: "encryptionkey@0"
I20260806 10:47:08.781579 17251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:47:08.785115 17266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:08.786394 17251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:08.786574 17251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "5a4c639026de458c812a7a79e70e3413"
format_stamp: "Formatted at 2026-08-06 10:47:08 on dist-test-slave-0b58"
server_key: "d2756605e59c3deca6d94a723a28a8a4"
server_key_iv: "a837035b74db3c9f4c86e4d7635725ab"
server_key_version: "encryptionkey@0"
I20260806 10:47:08.786783 17251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:08.803877 17251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:08.808307 17251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:08.808593 17251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:08.809510 17251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:08.810832 17251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:08.810936 17251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:08.811093 17251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:08.811167 17251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:08.844558 17251 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:44263
I20260806 10:47:08.844609 17381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:44263 every 8 connection(s)
I20260806 10:47:08.845829 17251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:08.848824 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17251
I20260806 10:47:08.849020 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:08.849493 16468 external_mini_cluster.cc:1428] Setting key f85f4c2fcfb617c68cf360581002828e
Aug 06 10:47:08 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:08.870517 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:08.848324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45365 (local address 127.16.21.62:46565)
0806 10:47:08.848550 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:08.848557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:08.850723 (+ 2166us) server_negotiation.cc:407] Connection header received
0806 10:47:08.853251 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:08.853255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:08.853323 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:08.853396 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:08.855529 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.856229 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.858785 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.859206 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.859316 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:08.863448 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:08.863457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:08.863465 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:08.863531 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:08.867199 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:08.867888 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:08.867896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:08.867902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:08.868073 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:08.868559 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:08.868567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:08.868573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:08.868955 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:08.869113 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:08.869765 (+ 652us) server_negotiation.cc:299] Negotiation successful
0806 10:47:08.870041 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 10:47:08.871001 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:08.848808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46565 (local address 127.16.21.2:45365)
0806 10:47:08.850547 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:08.850601 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:08.852960 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:08.853673 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:08.853697 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:08.854514 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:08.855352 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:08.855379 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.856453 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.856460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:08.858561 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:08.858578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:08.859446 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:08.859457 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:08.859713 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:08.860479 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:08.860522 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:08.863197 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:08.867336 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:08.867342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:08.867352 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:08.867761 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:08.868197 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:08.868200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:08.868202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:08.868431 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:08.869086 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:08.869092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:08.869365 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 10:47:08.869567 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 10:47:08.869941 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"thread_start_us":1224,"threads_started":1}
I20260806 10:47:08.872452 17383 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46565
I20260806 10:47:08.872812 17383 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
I20260806 10:47:08.873713 17383 heartbeater.cc:511] Master 127.16.21.62:46565 requested a full tablet report, sending...
I20260806 10:47:08.875147 17054 ts_manager.cc:194] Registered new tserver with Master: 5a4c639026de458c812a7a79e70e3413 (127.16.21.2:44263)
I20260806 10:47:08.876072 17054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:45365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:08.899524 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46565
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:09.218848 17388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:09.219106 17388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:09.219275 17388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:09.226414 17388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:09.226512 17388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:09.226706 17388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:09.235957 17388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:09.237130 17388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:09.238452 17388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:09.251456 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:09.251545 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:09.251510 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:09.252851 17388 server_base.cc:1034] running on GCE node
I20260806 10:47:09.253383 17388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:09.254216 17388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:09.255434 17388 hybrid_clock.cc:648] HybridClock initialized: now 1786013229255379 us; error 30 us; skew 500 ppm
I20260806 10:47:09.255743 17388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:09.258880 17388 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:09.260156 17388 webserver.cc:460] Webserver started at http://127.16.21.3:40299/ using document root <none> and password file <none>
I20260806 10:47:09.260968 17388 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:09.261070 17388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:09.261327 17388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:09.264406 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "945ba3f99c524ff9bc12005f44ae53e6"
format_stamp: "Formatted at 2026-08-06 10:47:09 on dist-test-slave-0b58"
server_key: "ab324a6e2c285abf93d9cbbcbd6d3ac5"
server_key_iv: "7fe63282656006ff29fcb56a1ab57f43"
server_key_version: "encryptionkey@0"
I20260806 10:47:09.265137 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "945ba3f99c524ff9bc12005f44ae53e6"
format_stamp: "Formatted at 2026-08-06 10:47:09 on dist-test-slave-0b58"
server_key: "ab324a6e2c285abf93d9cbbcbd6d3ac5"
server_key_iv: "7fe63282656006ff29fcb56a1ab57f43"
server_key_version: "encryptionkey@0"
I20260806 10:47:09.270089 17388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:47:09.273907 17403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:09.275147 17388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:47:09.275332 17388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "945ba3f99c524ff9bc12005f44ae53e6"
format_stamp: "Formatted at 2026-08-06 10:47:09 on dist-test-slave-0b58"
server_key: "ab324a6e2c285abf93d9cbbcbd6d3ac5"
server_key_iv: "7fe63282656006ff29fcb56a1ab57f43"
server_key_version: "encryptionkey@0"
I20260806 10:47:09.275527 17388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:09.308516 17388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:09.312966 17388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:09.313287 17388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:09.314249 17388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:09.315573 17388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:09.315680 17388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:09.315835 17388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:09.315917 17388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:09.350944 17388 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:36901
I20260806 10:47:09.351029 17517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:36901 every 8 connection(s)
I20260806 10:47:09.352141 17388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:09.357836 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17388
I20260806 10:47:09.358062 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:09.358511 16468 external_mini_cluster.cc:1428] Setting key 8118604406027095b9f3e196974710ef
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:09.372591 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.354650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:43993 (local address 127.16.21.62:46565)
0806 10:47:09.354838 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.354843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.355920 (+ 1077us) server_negotiation.cc:407] Connection header received
0806 10:47:09.358018 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.358025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.358125 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.358237 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.360006 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.361085 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.362817 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.363302 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.363396 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.366680 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.366685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.366688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.366724 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.369572 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.370273 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.370276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.370287 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.370420 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.370829 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.370833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.370835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.371131 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.371262 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.371962 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.372183 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:47:09.373286 17244 heartbeater.cc:503] Master 127.16.21.62:46565 was elected leader, sending a full tablet report...
I20260806 10:47:09.373759 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.355107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46565 (local address 127.16.21.3:43993)
0806 10:47:09.355768 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:09.355805 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:09.357676 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:09.358472 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:09.358480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:09.359064 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:09.359829 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.359856 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.361252 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.361256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.362677 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.362691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.363494 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.363499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.363684 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.364329 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:09.364359 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:09.366485 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:09.369714 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:09.369721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:09.369733 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:09.370153 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:09.370520 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:09.370524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:09.370526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:09.370697 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:09.371251 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:09.371256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:09.371553 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 10:47:09.371787 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 10:47:09.372272 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":282,"threads_started":1}
I20260806 10:47:09.375204 17518 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46565
I20260806 10:47:09.375466 17518 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:09.376114 17518 heartbeater.cc:511] Master 127.16.21.62:46565 requested a full tablet report, sending...
I20260806 10:47:09.377537 17053 ts_manager.cc:194] Registered new tserver with Master: 945ba3f99c524ff9bc12005f44ae53e6 (127.16.21.3:36901)
I20260806 10:47:09.378388 17053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:43993
I20260806 10:47:09.386750 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:47:09.406251 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.393186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.16.21.62:46565)
0806 10:47:09.393351 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.393356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.393835 (+ 479us) server_negotiation.cc:407] Connection header received
0806 10:47:09.394074 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.394078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.394147 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.394271 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.395765 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.396583 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.397898 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.398311 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.398458 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.400866 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.400872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.400878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.400915 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.403543 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.404194 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.404201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.404205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.404405 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.404781 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.404788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.404793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.405103 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.405382 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.405711 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.405931 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:47:09.412148 17053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:09.419016 17053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:09.452641 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.431872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.16.21.1:41303)
0806 10:47:09.444790 (+ 12918us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.444798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.444826 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:47:09.444939 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.444947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.445143 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.445360 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:09.445949 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.447584 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.449660 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.450926 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.451050 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.451296 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.452096 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.452291 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12755,"thread_start_us":12588,"threads_started":1}
I20260806 10:47:09.452880 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.430586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:36901 (local address 127.0.0.1:48586)
0806 10:47:09.431610 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:09.432892 (+ 1282us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:09.433118 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:09.444919 (+ 11801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:09.444926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:09.444955 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:09.445402 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.445419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.447448 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.447453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.450042 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.450058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.451843 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.451850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.452031 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.452063 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:47:09.452146 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 10:47:09.452333 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":34,"thread_start_us":549,"threads_started":1}
I20260806 10:47:09.453105 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.431078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.16.21.3:36901)
0806 10:47:09.444066 (+ 12988us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.444079 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.444105 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:47:09.444180 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.444185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.444463 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.444904 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:09.445657 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.447278 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.450219 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.451619 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.451747 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.451981 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.452573 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.452742 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12790,"thread_start_us":13512,"threads_started":1}
I20260806 10:47:09.452476 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.431099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:41303 (local address 127.0.0.1:35216)
0806 10:47:09.442845 (+ 11746us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:09.442868 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:09.443015 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:09.445360 (+ 2345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:09.445368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:09.445391 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:09.445781 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.445791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.447739 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.447745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.449496 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.449512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.451438 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.451446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.451635 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.451705 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 10:47:09.451837 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:47:09.452046 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11533,"thread_start_us":12010,"threads_started":1}
I20260806 10:47:09.453752 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.433769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:44263 (local address 127.0.0.1:34390)
0806 10:47:09.443349 (+ 9580us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:09.443366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:09.443479 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:09.444786 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:09.444791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:09.444815 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:09.446218 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.446233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.448015 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.448020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.450271 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.450281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.452346 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.452352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.452542 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.452574 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:47:09.452666 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 10:47:09.453504 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9449,"mutex_wait_us":837,"thread_start_us":9298,"threads_started":1}
I20260806 10:47:09.454327 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.434366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.16.21.2:44263)
0806 10:47:09.443767 (+ 9401us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.443791 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.443833 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:47:09.443965 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.443977 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.444308 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.444541 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:09.446404 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.447870 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.450412 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.451866 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.451990 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.452173 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.452890 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.453142 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9214,"thread_start_us":9080,"threads_started":1}
I20260806 10:47:09.457420 17315 tablet_service.cc:1467] Processing CreateTablet for tablet e19a5f2658f34b7d8358c4d01648b63e (DEFAULT_TABLE table=test-table [id=ce8267909abe4540890a54c8e53fac36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:09.457311 17179 tablet_service.cc:1467] Processing CreateTablet for tablet e19a5f2658f34b7d8358c4d01648b63e (DEFAULT_TABLE table=test-table [id=ce8267909abe4540890a54c8e53fac36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:09.457855 17452 tablet_service.cc:1467] Processing CreateTablet for tablet e19a5f2658f34b7d8358c4d01648b63e (DEFAULT_TABLE table=test-table [id=ce8267909abe4540890a54c8e53fac36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:09.458678 17315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19a5f2658f34b7d8358c4d01648b63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:09.458784 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19a5f2658f34b7d8358c4d01648b63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:09.459136 17452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19a5f2658f34b7d8358c4d01648b63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:09.471817 17536 tablet_bootstrap.cc:492] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: Bootstrap starting.
I20260806 10:47:09.473877 17537 tablet_bootstrap.cc:492] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: Bootstrap starting.
I20260806 10:47:09.474822 17538 tablet_bootstrap.cc:492] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: Bootstrap starting.
I20260806 10:47:09.476289 17536 tablet_bootstrap.cc:654] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:09.477972 17537 tablet_bootstrap.cc:654] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:09.478222 17538 tablet_bootstrap.cc:654] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:09.478222 17536 log.cc:824] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:09.479245 17537 log.cc:824] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:09.479466 17538 log.cc:824] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:09.481626 17536 tablet_bootstrap.cc:492] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: No bootstrap required, opened a new log
I20260806 10:47:09.481906 17536 ts_tablet_manager.cc:1397] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260806 10:47:09.482797 17538 tablet_bootstrap.cc:492] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: No bootstrap required, opened a new log
I20260806 10:47:09.483074 17538 ts_tablet_manager.cc:1397] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260806 10:47:09.490516 17537 tablet_bootstrap.cc:492] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: No bootstrap required, opened a new log
I20260806 10:47:09.490816 17537 ts_tablet_manager.cc:1397] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260806 10:47:09.493304 17536 raft_consensus.cc:348] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.493750 17536 raft_consensus.cc:374] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:09.493865 17536 raft_consensus.cc:729] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4c639026de458c812a7a79e70e3413, State: Initialized, Role: FOLLOWER
I20260806 10:47:09.493871 17538 raft_consensus.cc:348] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.494283 17538 raft_consensus.cc:374] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:09.494379 17538 raft_consensus.cc:729] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945ba3f99c524ff9bc12005f44ae53e6, State: Initialized, Role: FOLLOWER
I20260806 10:47:09.494520 17536 consensus_queue.cc:260] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.494946 17538 consensus_queue.cc:260] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.495731 17383 heartbeater.cc:503] Master 127.16.21.62:46565 was elected leader, sending a full tablet report...
I20260806 10:47:09.496532 17536 ts_tablet_manager.cc:1428] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260806 10:47:09.499694 17537 raft_consensus.cc:348] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.500028 17537 raft_consensus.cc:374] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:09.500156 17537 raft_consensus.cc:729] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac0fc1e9f574d1e9d656c788538cbdc, State: Initialized, Role: FOLLOWER
I20260806 10:47:09.500789 17537 consensus_queue.cc:260] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.502664 17537 ts_tablet_manager.cc:1428] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 10:47:09.502877 17538 ts_tablet_manager.cc:1428] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: Time spent starting tablet: real 0.020s user 0.015s sys 0.006s
I20260806 10:47:09.503299 17518 heartbeater.cc:503] Master 127.16.21.62:46565 was elected leader, sending a full tablet report...
I20260806 10:47:09.551298 17543 raft_consensus.cc:482] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:09.551589 17543 raft_consensus.cc:504] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.553303 17543 leader_election.cc:288] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ac0fc1e9f574d1e9d656c788538cbdc (127.16.21.1:41303), 5a4c639026de458c812a7a79e70e3413 (127.16.21.2:44263)
I20260806 10:47:09.559902 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.553551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:41303 (local address 127.16.21.3:36083)
0806 10:47:09.553808 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:09.553832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:09.553968 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:09.554479 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:09.554484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:09.554523 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:09.554748 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.554760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.556088 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.556092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.558041 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.558058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.559371 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.559375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.559506 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.559530 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:47:09.559603 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 10:47:09.559692 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":4}
I20260806 10:47:09.560037 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.553716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:36083 (local address 127.16.21.1:41303)
0806 10:47:09.553943 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.553948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.553970 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:47:09.554211 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.554220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.554311 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.554405 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:09.554917 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.555949 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.558208 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.559210 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.559358 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.559465 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.559659 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.559789 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 10:47:09.560985 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.553979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:33855 (local address 127.16.21.2:44263)
0806 10:47:09.554167 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.554175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.554739 (+ 564us) server_negotiation.cc:407] Connection header received
0806 10:47:09.554917 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.554922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.555002 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.555098 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:09.555580 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.556522 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.558680 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.559892 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.560020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.560124 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.560594 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.560727 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 10:47:09.561117 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.553728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:44263 (local address 127.16.21.3:33855)
0806 10:47:09.554622 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:09.554640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:09.554761 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:09.555126 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:09.555129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:09.555149 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:09.555426 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.555435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.556677 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.556683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.558533 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:09.558544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.560278 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.560287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:09.560434 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.560457 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:47:09.560568 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:47:09.560700 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":23,"thread_start_us":505,"threads_started":1}
I20260806 10:47:09.560984 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19a5f2658f34b7d8358c4d01648b63e" candidate_uuid: "945ba3f99c524ff9bc12005f44ae53e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" is_pre_election: true
I20260806 10:47:09.561622 17199 raft_consensus.cc:2454] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945ba3f99c524ff9bc12005f44ae53e6 in term 0.
I20260806 10:47:09.562134 17335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19a5f2658f34b7d8358c4d01648b63e" candidate_uuid: "945ba3f99c524ff9bc12005f44ae53e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4c639026de458c812a7a79e70e3413" is_pre_election: true
I20260806 10:47:09.562373 17407 leader_election.cc:302] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac0fc1e9f574d1e9d656c788538cbdc, 945ba3f99c524ff9bc12005f44ae53e6; no voters:
I20260806 10:47:09.562608 17335 raft_consensus.cc:2454] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945ba3f99c524ff9bc12005f44ae53e6 in term 0.
I20260806 10:47:09.562762 17543 raft_consensus.cc:2781] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:09.562927 17543 raft_consensus.cc:482] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:09.563120 17543 raft_consensus.cc:3037] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:09.565567 17543 raft_consensus.cc:504] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.566262 17543 leader_election.cc:288] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6ac0fc1e9f574d1e9d656c788538cbdc (127.16.21.1:41303), 5a4c639026de458c812a7a79e70e3413 (127.16.21.2:44263)
I20260806 10:47:09.566684 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19a5f2658f34b7d8358c4d01648b63e" candidate_uuid: "945ba3f99c524ff9bc12005f44ae53e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc"
I20260806 10:47:09.566813 17335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19a5f2658f34b7d8358c4d01648b63e" candidate_uuid: "945ba3f99c524ff9bc12005f44ae53e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4c639026de458c812a7a79e70e3413"
I20260806 10:47:09.566895 17199 raft_consensus.cc:3037] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:09.567047 17335 raft_consensus.cc:3037] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:09.569258 17199 raft_consensus.cc:2454] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945ba3f99c524ff9bc12005f44ae53e6 in term 1.
I20260806 10:47:09.569430 17335 raft_consensus.cc:2454] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945ba3f99c524ff9bc12005f44ae53e6 in term 1.
I20260806 10:47:09.569682 17407 leader_election.cc:302] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac0fc1e9f574d1e9d656c788538cbdc, 945ba3f99c524ff9bc12005f44ae53e6; no voters:
I20260806 10:47:09.569978 17543 raft_consensus.cc:2781] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:09.570427 17543 raft_consensus.cc:686] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [term 1 LEADER]: Becoming Leader. State: Replica: 945ba3f99c524ff9bc12005f44ae53e6, State: Running, Role: LEADER
I20260806 10:47:09.571199 17543 consensus_queue.cc:237] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } }
I20260806 10:47:09.576035 17054 catalog_manager.cc:5495] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 945ba3f99c524ff9bc12005f44ae53e6 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "945ba3f99c524ff9bc12005f44ae53e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945ba3f99c524ff9bc12005f44ae53e6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ac0fc1e9f574d1e9d656c788538cbdc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4c639026de458c812a7a79e70e3413" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 44263 } health_report { overall_health: UNKNOWN } } }
W20260806 10:47:09.595031 17245 tablet.cc:2369] T e19a5f2658f34b7d8358c4d01648b63e P 6ac0fc1e9f574d1e9d656c788538cbdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:09.597903 17384 tablet.cc:2369] T e19a5f2658f34b7d8358c4d01648b63e P 5a4c639026de458c812a7a79e70e3413: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:09.603763 17519 tablet.cc:2369] T e19a5f2658f34b7d8358c4d01648b63e P 945ba3f99c524ff9bc12005f44ae53e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:09.612586 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.596082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.16.21.1:41303)
0806 10:47:09.596307 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.596314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.596952 (+ 638us) server_negotiation.cc:407] Connection header received
0806 10:47:09.597083 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.597087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.597327 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.597450 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.599705 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.600704 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.602066 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.602525 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.602671 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.606849 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.606863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.606877 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.606909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.609753 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.610432 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.610440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.610444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.610584 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.610989 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.610994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.610997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.611366 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.611675 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.611983 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.612205 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:09.663872 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.650398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.16.21.1:41303)
0806 10:47:09.650668 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.650675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.650889 (+ 214us) server_negotiation.cc:407] Connection header received
0806 10:47:09.651025 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.651029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.651094 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.651177 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.652560 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.653294 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.654508 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.654921 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.655011 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.658899 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.658904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.658907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.658951 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.661460 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.662010 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.662013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.662015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.662167 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.662500 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.662503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.662505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.662897 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.663140 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.663385 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.663546 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:09.713696 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.700753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.16.21.1:41303)
0806 10:47:09.701016 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.701024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.701184 (+ 160us) server_negotiation.cc:407] Connection header received
0806 10:47:09.701339 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.701343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.701406 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.701491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.702814 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.703498 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.704824 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.705245 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.705348 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.709076 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.709080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.709083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.709113 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.711428 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.711980 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.711983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.711985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.712095 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.712494 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.712497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.712499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.712768 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.712962 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.713138 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.713334 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260806 10:47:09.715457 17159 tablet_service.cc:2908] Rejecting scan request for tablet e19a5f2658f34b7d8358c4d01648b63e: Uninitialized: clean time has not yet been initialized
I20260806 10:47:09.719244 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17115
I20260806 10:47:09.731575 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.720026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.16.21.1:41303)
0806 10:47:09.720218 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.720226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.720660 (+ 434us) server_negotiation.cc:407] Connection header received
0806 10:47:09.720786 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.720789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.720851 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.720927 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.722042 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.722680 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.723833 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.724334 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.724442 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.726573 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.726577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.726580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.726612 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.729028 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.729739 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.729743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.729746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.729876 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.730240 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.730243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.730245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.730500 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.730714 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.731005 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.731204 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:47:09.732175 17239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:09.825538 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17115
I20260806 10:47:09.846362 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17251
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:09.861896 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.847452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.16.21.2:44263)
0806 10:47:09.847639 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.847646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.847685 (+ 39us) server_negotiation.cc:407] Connection header received
0806 10:47:09.847911 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.847921 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.847984 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.848111 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.849458 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.850237 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.851556 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.851990 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.852147 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.856322 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.856333 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.856346 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.856397 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.859535 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.860114 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.860120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.860122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.860241 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.860661 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.860664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.860666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.861035 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.861277 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.861497 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.861654 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:47:09.862547 17377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:09.950147 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17251
I20260806 10:47:09.970368 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17388
Aug 06 10:47:09 dist-test-slave-0b58 krb5kdc[17008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:09.985523 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:09.971211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.16.21.3:36901)
0806 10:47:09.971655 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:09.971659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:09.971675 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:47:09.971729 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:09.971733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:09.971801 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:09.971890 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:09.973228 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.974065 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.975286 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:09.975660 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:09.975748 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:09.979910 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:09.979932 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:09.979954 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:09.980000 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:09.982697 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.983298 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.983304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.983306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.983419 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:09.983863 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:09.983870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:09.983873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:09.984229 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:09.984586 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:09.984895 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 10:47:09.985137 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
I20260806 10:47:09.986218 17512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:10.076762 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:10.075508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:44263 (local address 127.16.21.3:35731)
0806 10:47:10.076354 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:10.076534 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.2:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":534,"threads_started":1}
I20260806 10:47:10.076814 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17388
I20260806 10:47:10.096630 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17021
I20260806 10:47:10.097793 17084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:10.210253 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17021
2026-08-06T10:47:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4217 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:10 dist-test-slave-0b58 krb5kdc[17568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:10 dist-test-slave-0b58 krb5kdc[17568](info): set up 2 sockets
Aug 06 10:47:10 dist-test-slave-0b58 krb5kdc[17568](info): commencing operation
krb5kdc: starting...
W20260806 10:47:11.303990 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:11 dist-test-slave-0b58 krb5kdc[17568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:11.497874 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35895
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:11.816212 17581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:11.816516 17581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:11.816696 17581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:11.823822 17581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:11.823939 17581 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 10:47:11.824013 17581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:11.824126 17581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:11.824214 17581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:11.824343 17581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:11.833897 17581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:11.835054 17581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:11.836433 17581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:11.847870 17586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:11.848053 17587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:11.847896 17589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:11.849807 17581 server_base.cc:1034] running on GCE node
I20260806 10:47:11.850564 17581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:11.851804 17581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:11.853049 17581 hybrid_clock.cc:648] HybridClock initialized: now 1786013231852978 us; error 28 us; skew 500 ppm
I20260806 10:47:11.853375 17581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:11 dist-test-slave-0b58 krb5kdc[17568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013231, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:11.856417 17581 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:11.857847 17581 webserver.cc:460] Webserver started at http://127.16.21.62:36665/ using document root <none> and password file <none>
I20260806 10:47:11.858579 17581 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:11.858726 17581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:11.859050 17581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:11.861912 17581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "4744c44d430b4d6e9cd6931017e5af83"
format_stamp: "Formatted at 2026-08-06 10:47:11 on dist-test-slave-0b58"
server_key: "1790b7a70106e01e265bc3c3ce9699b1"
server_key_iv: "acc79d5249e0c95b3f65930b6b25a6a8"
server_key_version: "encryptionkey@0"
I20260806 10:47:11.862700 17581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "4744c44d430b4d6e9cd6931017e5af83"
format_stamp: "Formatted at 2026-08-06 10:47:11 on dist-test-slave-0b58"
server_key: "1790b7a70106e01e265bc3c3ce9699b1"
server_key_iv: "acc79d5249e0c95b3f65930b6b25a6a8"
server_key_version: "encryptionkey@0"
I20260806 10:47:11.867651 17581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:47:11.870944 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:11.872104 17581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 10:47:11.872349 17581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "4744c44d430b4d6e9cd6931017e5af83"
format_stamp: "Formatted at 2026-08-06 10:47:11 on dist-test-slave-0b58"
server_key: "1790b7a70106e01e265bc3c3ce9699b1"
server_key_iv: "acc79d5249e0c95b3f65930b6b25a6a8"
server_key_version: "encryptionkey@0"
I20260806 10:47:11.872562 17581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:11.899765 17581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:11.903715 17581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:11.904006 17581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:11.926015 17581 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35895
I20260806 10:47:11.926034 17659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35895 every 8 connection(s)
I20260806 10:47:11.927320 17581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:11.930907 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:11.934306 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17581
I20260806 10:47:11.934567 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:11.935428 16468 external_mini_cluster.cc:1428] Setting key 3dba9d8d2b2cca340c71e9e9e4bcb39b
I20260806 10:47:11.943884 17660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: Bootstrap starting.
I20260806 10:47:11.947618 17660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:11.948939 17660 log.cc:824] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:47:11 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:11.951730 17660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: No bootstrap required, opened a new log
I20260806 10:47:11.958298 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:11.937462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.16.21.62:35895)
0806 10:47:11.938228 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:11.938238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:11.938293 (+ 55us) server_negotiation.cc:407] Connection header received
0806 10:47:11.939978 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:11.939997 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:11.940438 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:11.940735 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:11.942416 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:11.943626 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:11.945361 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:11.946016 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:11.946132 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:11.951094 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:11.951101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:11.951113 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:11.951149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:11.954371 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:11.955061 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:11.955070 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:11.955082 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:11.955264 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:11.955662 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:11.955666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:11.955668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:11.956177 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:11.956559 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:11.956943 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 10:47:11.957325 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":394,"threads_started":1}
I20260806 10:47:11.959857 17660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744c44d430b4d6e9cd6931017e5af83" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35895 } }
I20260806 10:47:11.960306 17660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:11.960413 17660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4744c44d430b4d6e9cd6931017e5af83, State: Initialized, Role: FOLLOWER
I20260806 10:47:11.960971 17660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744c44d430b4d6e9cd6931017e5af83" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35895 } }
I20260806 10:47:11.961196 17660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:11.961359 17660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:11.961525 17660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:11.963552 17660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744c44d430b4d6e9cd6931017e5af83" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35895 } }
I20260806 10:47:11.964133 17660 leader_election.cc:302] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4744c44d430b4d6e9cd6931017e5af83; no voters:
I20260806 10:47:11.964732 17660 leader_election.cc:288] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:11.964880 17665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:11.965608 17665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [term 1 LEADER]: Becoming Leader. State: Replica: 4744c44d430b4d6e9cd6931017e5af83, State: Running, Role: LEADER
I20260806 10:47:11.966094 17665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744c44d430b4d6e9cd6931017e5af83" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35895 } }
I20260806 10:47:11.966738 17660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:11.969290 17666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4744c44d430b4d6e9cd6931017e5af83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744c44d430b4d6e9cd6931017e5af83" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35895 } } }
I20260806 10:47:11.969301 17667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4744c44d430b4d6e9cd6931017e5af83. Latest consensus state: current_term: 1 leader_uuid: "4744c44d430b4d6e9cd6931017e5af83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4744c44d430b4d6e9cd6931017e5af83" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35895 } } }
I20260806 10:47:11.969751 17666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:11.969974 17667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:11.971771 17671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:11.978351 17671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:11.986996 17671 catalog_manager.cc:1347] Generated new cluster ID: f4c1a0ed09ef4a4388b853655f80e255
I20260806 10:47:11.987150 17671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:11.999424 17671 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:12.001238 17671 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:12.011915 17671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: Generated new TSK 0
I20260806 10:47:12.012882 17671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:12.064036 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35895
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:12.397830 17686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:12.398083 17686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:12.398180 17686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:12.405264 17686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:12.405354 17686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:12.405436 17686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:47:12.405514 17686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:12.414758 17686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:12.415830 17686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:12.417048 17686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:12.429409 17691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:12.429448 17692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:12.429476 17694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:12.430596 17686 server_base.cc:1034] running on GCE node
I20260806 10:47:12.431157 17686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:12.432010 17686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:12.433261 17686 hybrid_clock.cc:648] HybridClock initialized: now 1786013232433181 us; error 55 us; skew 500 ppm
I20260806 10:47:12.433537 17686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:12 dist-test-slave-0b58 krb5kdc[17568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:12.436631 17686 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:12.437963 17686 webserver.cc:460] Webserver started at http://127.16.21.1:42305/ using document root <none> and password file <none>
I20260806 10:47:12.438604 17686 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:12.438733 17686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:12.439046 17686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:12.441737 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "0be832b12afd46fb8c15e2036e30d841"
format_stamp: "Formatted at 2026-08-06 10:47:12 on dist-test-slave-0b58"
server_key: "70e8f994fd6c6cc382fcb2f2d7581d70"
server_key_iv: "304e070ad70d1b8ab2557053904ff25e"
server_key_version: "encryptionkey@0"
I20260806 10:47:12.442514 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "0be832b12afd46fb8c15e2036e30d841"
format_stamp: "Formatted at 2026-08-06 10:47:12 on dist-test-slave-0b58"
server_key: "70e8f994fd6c6cc382fcb2f2d7581d70"
server_key_iv: "304e070ad70d1b8ab2557053904ff25e"
server_key_version: "encryptionkey@0"
I20260806 10:47:12.447402 17686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 10:47:12.450944 17701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:12.452334 17686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:47:12.452498 17686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "0be832b12afd46fb8c15e2036e30d841"
format_stamp: "Formatted at 2026-08-06 10:47:12 on dist-test-slave-0b58"
server_key: "70e8f994fd6c6cc382fcb2f2d7581d70"
server_key_iv: "304e070ad70d1b8ab2557053904ff25e"
server_key_version: "encryptionkey@0"
I20260806 10:47:12.452704 17686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:12.470791 17686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:12.474922 17686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:12.475184 17686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:12.477998 17686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:12.478104 17686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:12.478282 17686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:12.478368 17686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:47:12 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:12.502735 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:12.479042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.16.21.1:44885)
0806 10:47:12.479880 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:12.479932 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:12.481962 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:12.482695 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:12.482702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:12.483446 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:12.484052 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:12.484068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.485822 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.485829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:12.488921 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:12.488946 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.489914 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.489922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:12.490156 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:12.490931 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:12.490955 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:12.494298 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:12.497572 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:12.497585 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:12.497602 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:12.498261 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:12.498771 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:12.498777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:12.498780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:12.499035 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:12.499719 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:12.499726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:12.500271 (+ 545us) client_negotiation.cc:769] Sending connection context
0806 10:47:12.500594 (+ 323us) client_negotiation.cc:240] Negotiation successful
0806 10:47:12.501101 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":381,"threads_started":1}
I20260806 10:47:12.503376 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:12.478523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:44885 (local address 127.16.21.62:35895)
0806 10:47:12.480612 (+ 2089us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:12.480619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:12.480642 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:47:12.482273 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:12.482282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:12.482374 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:12.482508 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:12.484253 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.485563 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.489103 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.489639 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.489787 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:12.494504 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:12.494516 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:12.494520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:12.494553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:12.497431 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:12.498414 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:12.498419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:12.498423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:12.498620 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:12.499164 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:12.499176 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:12.499179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:12.499582 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:12.499756 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:12.502810 (+ 3054us) server_negotiation.cc:299] Negotiation successful
0806 10:47:12.503061 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1948,"thread_start_us":1841,"threads_started":1}
I20260806 10:47:12.525141 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:12.510262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.16.21.1:45243)
0806 10:47:12.510849 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:12.510867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:12.511014 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:12.511374 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:12.511379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:12.511808 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:12.512172 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:12.512194 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.513453 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.513458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:12.515216 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:12.515235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.516012 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.516018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:12.516192 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:12.516914 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:12.516941 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:12.518058 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:12.522439 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:12.522449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:12.522454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:12.522816 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:12.523284 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:12.523287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:12.523289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:12.523404 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:12.524098 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:12.524108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:12.524338 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 10:47:12.524533 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 10:47:12.524827 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":343,"threads_started":1}
I20260806 10:47:12.525717 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:12.510220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:45243 (local address 127.16.21.62:35895)
0806 10:47:12.510374 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:12.510378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:12.511138 (+ 760us) server_negotiation.cc:407] Connection header received
0806 10:47:12.511181 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:12.511184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:12.511238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:12.511350 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:12.512372 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.513324 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.515425 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.515828 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.515967 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:12.518266 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:12.518275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:12.518282 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:12.518330 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:12.522284 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:12.522943 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:12.522950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:12.522956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:12.523167 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:12.523508 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:12.523516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:12.523522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:12.523955 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:12.524131 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:12.524659 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 10:47:12.524978 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:47:12.529331 17686 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:41721
I20260806 10:47:12.529368 17825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:41721 every 8 connection(s)
I20260806 10:47:12.530524 17686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:12.532359 17826 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35895
I20260806 10:47:12.532547 17826 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:12.533113 17826 heartbeater.cc:511] Master 127.16.21.62:35895 requested a full tablet report, sending...
I20260806 10:47:12.534660 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17686
I20260806 10:47:12.534925 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:12.535220 17613 ts_manager.cc:194] Registered new tserver with Master: 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721)
I20260806 10:47:12.535765 16468 external_mini_cluster.cc:1428] Setting key 5ac2d3bed74646e9a8d698d8fd72375a
I20260806 10:47:12.537150 17613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:44885
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:12.581422 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35895
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:12.901742 17830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:12.902030 17830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:12.902209 17830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:12.909641 17830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:12.909790 17830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:12.909930 17830 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:47:12.910066 17830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:12.919085 17830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:12.920207 17830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:12.921468 17830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:12.934692 17835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:12.934783 17836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:12.935309 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:12.935719 17830 server_base.cc:1034] running on GCE node
I20260806 10:47:12.936259 17830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:12.937273 17830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:12.938520 17830 hybrid_clock.cc:648] HybridClock initialized: now 1786013232938454 us; error 36 us; skew 500 ppm
I20260806 10:47:12.938822 17830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:12 dist-test-slave-0b58 krb5kdc[17568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:12.947222 17830 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:12.948717 17830 webserver.cc:460] Webserver started at http://127.16.21.2:33139/ using document root <none> and password file <none>
I20260806 10:47:12.949369 17830 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:12.949499 17830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:12.949821 17830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:12.952695 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "c1f74fa4300748f58f4575217f13cd42"
format_stamp: "Formatted at 2026-08-06 10:47:12 on dist-test-slave-0b58"
server_key: "bd8969eacc927b61c8927c4c242e28df"
server_key_iv: "c004b5fac6344f23d93fdbc2bfd8cde3"
server_key_version: "encryptionkey@0"
I20260806 10:47:12.953440 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "c1f74fa4300748f58f4575217f13cd42"
format_stamp: "Formatted at 2026-08-06 10:47:12 on dist-test-slave-0b58"
server_key: "bd8969eacc927b61c8927c4c242e28df"
server_key_iv: "c004b5fac6344f23d93fdbc2bfd8cde3"
server_key_version: "encryptionkey@0"
I20260806 10:47:12.958734 17830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:47:12.962139 17845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:12.963344 17830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 10:47:12.963537 17830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "c1f74fa4300748f58f4575217f13cd42"
format_stamp: "Formatted at 2026-08-06 10:47:12 on dist-test-slave-0b58"
server_key: "bd8969eacc927b61c8927c4c242e28df"
server_key_iv: "c004b5fac6344f23d93fdbc2bfd8cde3"
server_key_version: "encryptionkey@0"
I20260806 10:47:12.963779 17830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:12.975059 17830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:12.979420 17830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:12.979691 17830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:12.982041 17830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:12.982179 17830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:12.982345 17830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:12.982429 17830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:47:12 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:13.004423 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:12.982821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:53747 (local address 127.16.21.62:35895)
0806 10:47:12.983048 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:12.983057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:12.984367 (+ 1310us) server_negotiation.cc:407] Connection header received
0806 10:47:12.986378 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:12.986384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:12.986508 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:12.986620 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:12.988910 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.989954 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.992686 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.993322 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.993455 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:12.997217 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:12.997222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:12.997225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:12.997254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:13.000872 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.001590 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.001598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.001604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.001778 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.002379 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.002385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.002390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.002781 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:13.002939 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.003802 (+ 863us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.004018 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:47:13.005578 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:12.983284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.16.21.2:53747)
0806 10:47:12.984183 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:12.984230 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:12.986085 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:12.986855 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:12.986867 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:12.987593 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:12.988701 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:12.988737 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.990101 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.990110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:12.992491 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:12.992514 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:12.993586 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:12.993594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:12.993826 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:12.994504 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:12.994525 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:12.996977 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:13.001025 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.001033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.001047 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.001458 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.001921 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.001926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.001931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.002250 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.002922 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:13.002929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:13.003431 (+ 502us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.003700 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.004227 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":378,"threads_started":1}
I20260806 10:47:13.029446 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.014759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58073 (local address 127.16.21.62:35895)
0806 10:47:13.014955 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.014960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.016332 (+ 1372us) server_negotiation.cc:407] Connection header received
0806 10:47:13.016569 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.016576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.016680 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.016798 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:13.017745 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.019147 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.020468 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.021121 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.021253 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.023040 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:13.023046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:13.023050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:13.023088 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:13.026147 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.027131 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.027134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.027136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.027258 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.027731 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.027733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.027735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.028017 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:13.028127 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.028795 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.029048 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:47:13.030218 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.014885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.16.21.2:58073)
0806 10:47:13.016201 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.016217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.016426 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.016868 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.016874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.017310 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:13.017598 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.017607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.019289 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.019293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.020300 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.020315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.021331 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.021339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.021518 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.022157 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:13.022172 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:13.022865 (+ 693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:13.026308 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.026316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.026323 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.027005 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.027373 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.027380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.027386 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.027588 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.028134 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:13.028141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:13.028441 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.028652 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.029084 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"thread_start_us":1009,"threads_started":1}
I20260806 10:47:13.041458 17830 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:36041
I20260806 10:47:13.041520 17967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:36041 every 8 connection(s)
I20260806 10:47:13.042572 17830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:13.044544 17968 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35895
I20260806 10:47:13.044766 17968 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:13.045382 17968 heartbeater.cc:511] Master 127.16.21.62:35895 requested a full tablet report, sending...
I20260806 10:47:13.046617 17613 ts_manager.cc:194] Registered new tserver with Master: c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041)
I20260806 10:47:13.047446 17613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:53747
I20260806 10:47:13.048822 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17830
I20260806 10:47:13.049037 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:13.049588 16468 external_mini_cluster.cc:1428] Setting key 97a343c0e6b8514be2b856660e0402f5
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:13.096150 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35895
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:13.417275 17972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:13.417621 17972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:13.417809 17972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:13.425007 17972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:13.425161 17972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:13.425316 17972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:47:13.425432 17972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:13.435192 17972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:13.436503 17972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:13.437808 17972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:13.451122 17977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:13.451076 17978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:13.450973 17980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:13.452085 17972 server_base.cc:1034] running on GCE node
I20260806 10:47:13.452703 17972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:13.453526 17972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:13.454772 17972 hybrid_clock.cc:648] HybridClock initialized: now 1786013233454707 us; error 36 us; skew 500 ppm
I20260806 10:47:13.455065 17972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:13 dist-test-slave-0b58 krb5kdc[17568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:13.458469 17972 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:13.459991 17972 webserver.cc:460] Webserver started at http://127.16.21.3:44727/ using document root <none> and password file <none>
I20260806 10:47:13.460741 17972 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:13.460872 17972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:13.461195 17972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:13.463974 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "d6fbdda51a5f42f38ebeb796d2c693f3"
format_stamp: "Formatted at 2026-08-06 10:47:13 on dist-test-slave-0b58"
server_key: "aaef801640915b0406def571a054e71f"
server_key_iv: "22f5a7940ebd52c721f8968fbc107df6"
server_key_version: "encryptionkey@0"
I20260806 10:47:13.464820 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "d6fbdda51a5f42f38ebeb796d2c693f3"
format_stamp: "Formatted at 2026-08-06 10:47:13 on dist-test-slave-0b58"
server_key: "aaef801640915b0406def571a054e71f"
server_key_iv: "22f5a7940ebd52c721f8968fbc107df6"
server_key_version: "encryptionkey@0"
I20260806 10:47:13.469938 17972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:13.473446 17987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:13.474713 17972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:47:13.474887 17972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "d6fbdda51a5f42f38ebeb796d2c693f3"
format_stamp: "Formatted at 2026-08-06 10:47:13 on dist-test-slave-0b58"
server_key: "aaef801640915b0406def571a054e71f"
server_key_iv: "22f5a7940ebd52c721f8968fbc107df6"
server_key_version: "encryptionkey@0"
I20260806 10:47:13.475095 17972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:13.502342 17972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:13.506680 17972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:13.506973 17972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:13.509855 17972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:13.509960 17972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:13.510147 17972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:13.510236 17972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:47:13 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:13.532294 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.510223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:48395 (local address 127.16.21.62:35895)
0806 10:47:13.510414 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.510423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.511772 (+ 1349us) server_negotiation.cc:407] Connection header received
0806 10:47:13.513838 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.513846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.513937 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.514048 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:13.516071 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.517397 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.519983 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.520940 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.521121 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.525808 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:13.525813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:13.525816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:13.525847 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:13.529071 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.529804 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.529809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.529813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.529966 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.530387 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.530393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.530396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.530796 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:13.530955 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.531668 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.531916 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:47:13.533207 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.510663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.16.21.3:48395)
0806 10:47:13.511587 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.511630 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.513538 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.514288 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.514296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.514999 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:13.515836 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.515884 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.517624 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.517631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.519786 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.519826 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.521168 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.521173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.521373 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.522081 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:13.522117 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:13.525556 (+ 3439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:13.529214 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.529223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.529235 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.529657 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.530084 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.530095 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.530098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.530268 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.530950 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:13.530957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:13.531251 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.531529 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.531982 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":405,"threads_started":1}
I20260806 10:47:13.539844 17826 heartbeater.cc:503] Master 127.16.21.62:35895 was elected leader, sending a full tablet report...
I20260806 10:47:13.555276 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.539799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.16.21.3:60179)
0806 10:47:13.540493 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.540519 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.540667 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.542090 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.542096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.542848 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:13.543384 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.543407 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.544766 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.544772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.546544 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.546557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.547325 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.547332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.547527 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.548145 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:13.548161 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:13.549177 (+ 1016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:13.552049 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.552054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.552058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.552554 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.552957 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:13.552965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:13.552968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:13.553166 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:13.553916 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:13.553921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:13.554164 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.554357 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.554661 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":792,"threads_started":1}
I20260806 10:47:13.556100 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.540117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:60179 (local address 127.16.21.62:35895)
0806 10:47:13.540294 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.540299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.540614 (+ 315us) server_negotiation.cc:407] Connection header received
0806 10:47:13.541658 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.541677 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.541770 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.541874 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:13.543563 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.544623 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.546699 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.547141 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.547290 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.549380 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:13.549390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:13.549394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:13.549429 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:13.551912 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.552694 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.552702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.552705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.552838 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.553325 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.553333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.553337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.553764 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:13.553929 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.555626 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.555882 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:47:13.572407 17972 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41201
I20260806 10:47:13.572480 18109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41201 every 8 connection(s)
I20260806 10:47:13.573707 17972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:13.575053 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 17972
I20260806 10:47:13.575263 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:13.575876 16468 external_mini_cluster.cc:1428] Setting key 80c5aa3c6abb712e2cf4df5b8a7ecd35
I20260806 10:47:13.575935 18110 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35895
I20260806 10:47:13.576184 18110 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:13.577041 18110 heartbeater.cc:511] Master 127.16.21.62:35895 requested a full tablet report, sending...
I20260806 10:47:13.578485 17612 ts_manager.cc:194] Registered new tserver with Master: d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3:41201)
I20260806 10:47:13.579566 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:48395
I20260806 10:47:13.591104 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:47:13 dist-test-slave-0b58 krb5kdc[17568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:47:13 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:13.641743 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.628682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.16.21.62:35895)
0806 10:47:13.628877 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.628881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.629113 (+ 232us) server_negotiation.cc:407] Connection header received
0806 10:47:13.629262 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.629266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.629332 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.629422 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:13.630766 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.631493 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.632770 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.633169 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.633259 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.637013 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:13.637017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:13.637020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:13.637049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:13.639533 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.640138 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.640142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.640144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.640262 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:13.640642 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:13.640645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:13.640647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:13.640920 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:13.641107 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.641327 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.641483 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:47:13.645687 17612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:13.652776 17612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:13.687188 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.665474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.16.21.3:41201)
0806 10:47:13.677745 (+ 12271us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.677755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.678941 (+ 1186us) server_negotiation.cc:407] Connection header received
0806 10:47:13.679045 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.679052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.679347 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.679583 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:13.680938 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.682653 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.684879 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.685960 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.686102 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.686383 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.686686 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.686851 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12057,"thread_start_us":12609,"threads_started":1}
I20260806 10:47:13.687274 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.665027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41201 (local address 127.0.0.1:38646)
0806 10:47:13.678640 (+ 13613us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.678658 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.678786 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.680068 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.680078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.680111 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:13.680641 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.680657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.682854 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.682858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.684708 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.684725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.686176 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.686183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.686455 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.686493 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.686627 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.686805 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13484,"mutex_wait_us":32,"thread_start_us":12943,"threads_started":1}
I20260806 10:47:13.688416 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.664932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:36041 (local address 127.0.0.1:58886)
0806 10:47:13.678519 (+ 13587us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.678571 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.678740 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.680269 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.680274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.680319 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:13.680848 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.680862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.682796 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.682802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.685046 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.685061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.687082 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.687088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.687716 (+ 628us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.687762 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.687859 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.687988 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13172,"mutex_wait_us":113,"thread_start_us":13067,"threads_started":1}
I20260806 10:47:13.688428 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.665180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.16.21.2:36041)
0806 10:47:13.677582 (+ 12402us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.677591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.678848 (+ 1257us) server_negotiation.cc:407] Connection header received
0806 10:47:13.678956 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.678964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.679235 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.679521 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:13.681034 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.682670 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.685210 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.686612 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.686730 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.686934 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.687929 (+ 995us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.688105 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12215,"thread_start_us":12340,"threads_started":1}
I20260806 10:47:13.689538 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.679028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:41721 (local address 127.0.0.1:50516)
0806 10:47:13.680434 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.680456 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.680614 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.681656 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.681661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.681689 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:13.682157 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.682169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.684368 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.684373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.686554 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.686570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.688735 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.688741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.688923 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.688955 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.689020 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.689140 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":50,"thread_start_us":1009,"threads_started":1}
I20260806 10:47:13.690220 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.679304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.16.21.1:41721)
0806 10:47:13.681178 (+ 1874us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.681184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.681204 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:47:13.681273 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.681278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.681416 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.682374 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:13.682602 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.684178 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.686749 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.688210 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.689525 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.689743 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.689816 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.689977 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1717,"thread_start_us":1587,"threads_started":1}
I20260806 10:47:13.692063 18043 tablet_service.cc:1467] Processing CreateTablet for tablet 91833ff577074e2bb0201e516f78e41b (DEFAULT_TABLE table=test-table [id=6b1a458f00ff41f5acf1a2569aef6f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:13.692380 17897 tablet_service.cc:1467] Processing CreateTablet for tablet 91833ff577074e2bb0201e516f78e41b (DEFAULT_TABLE table=test-table [id=6b1a458f00ff41f5acf1a2569aef6f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:13.693524 17897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91833ff577074e2bb0201e516f78e41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:13.693583 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91833ff577074e2bb0201e516f78e41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:13.693627 17754 tablet_service.cc:1467] Processing CreateTablet for tablet 91833ff577074e2bb0201e516f78e41b (DEFAULT_TABLE table=test-table [id=6b1a458f00ff41f5acf1a2569aef6f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:13.694947 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91833ff577074e2bb0201e516f78e41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:13.704349 18129 tablet_bootstrap.cc:492] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Bootstrap starting.
I20260806 10:47:13.707702 18129 tablet_bootstrap.cc:654] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:13.709015 18129 log.cc:824] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:13.709060 18131 tablet_bootstrap.cc:492] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Bootstrap starting.
I20260806 10:47:13.712958 18130 tablet_bootstrap.cc:492] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Bootstrap starting.
I20260806 10:47:13.713843 18131 tablet_bootstrap.cc:654] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:13.715413 18131 log.cc:824] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:13.716325 18130 tablet_bootstrap.cc:654] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:13.717471 18130 log.cc:824] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:13.718624 18129 tablet_bootstrap.cc:492] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: No bootstrap required, opened a new log
I20260806 10:47:13.718905 18129 ts_tablet_manager.cc:1397] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260806 10:47:13.719249 18131 tablet_bootstrap.cc:492] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: No bootstrap required, opened a new log
I20260806 10:47:13.719509 18131 ts_tablet_manager.cc:1397] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 10:47:13.719969 18130 tablet_bootstrap.cc:492] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: No bootstrap required, opened a new log
I20260806 10:47:13.720206 18130 ts_tablet_manager.cc:1397] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 10:47:13.727562 18130 raft_consensus.cc:348] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.727924 18130 raft_consensus.cc:374] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:13.728057 18130 raft_consensus.cc:729] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be832b12afd46fb8c15e2036e30d841, State: Initialized, Role: FOLLOWER
I20260806 10:47:13.728708 18130 consensus_queue.cc:260] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.729745 18129 raft_consensus.cc:348] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.730183 18129 raft_consensus.cc:374] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:13.730299 18129 raft_consensus.cc:729] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f74fa4300748f58f4575217f13cd42, State: Initialized, Role: FOLLOWER
I20260806 10:47:13.730422 18130 ts_tablet_manager.cc:1428] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260806 10:47:13.730878 18129 consensus_queue.cc:260] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.731215 18131 raft_consensus.cc:348] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.731658 18131 raft_consensus.cc:374] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:13.731770 18131 raft_consensus.cc:729] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6fbdda51a5f42f38ebeb796d2c693f3, State: Initialized, Role: FOLLOWER
I20260806 10:47:13.732110 17968 heartbeater.cc:503] Master 127.16.21.62:35895 was elected leader, sending a full tablet report...
I20260806 10:47:13.732374 18131 consensus_queue.cc:260] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.733665 18110 heartbeater.cc:503] Master 127.16.21.62:35895 was elected leader, sending a full tablet report...
I20260806 10:47:13.734139 18131 ts_tablet_manager.cc:1428] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260806 10:47:13.735287 18129 ts_tablet_manager.cc:1428] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260806 10:47:13.782835 17827 tablet.cc:2369] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:13.794528 17969 tablet.cc:2369] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:13.825405 18111 tablet.cc:2369] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:13.882332 18138 raft_consensus.cc:482] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:13.882644 18138 raft_consensus.cc:504] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.884522 18138 leader_election.cc:288] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041), 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721)
I20260806 10:47:13.890465 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.884893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:46711 (local address 127.16.21.2:36041)
0806 10:47:13.885106 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.885113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.885253 (+ 140us) server_negotiation.cc:407] Connection header received
0806 10:47:13.885467 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.885474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.885574 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.885682 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:13.886096 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.887071 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.888786 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.889549 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.889689 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.889802 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.890080 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.890207 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:47:13.890465 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.884954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:36041 (local address 127.16.21.3:46711)
0806 10:47:13.885137 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.885159 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.885326 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.885716 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.885720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.885756 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:13.885930 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.885939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.887259 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.887263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.888602 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.888615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.889714 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.889719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.889904 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.889928 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.890049 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.890180 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 10:47:13.891671 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.885072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:53847 (local address 127.16.21.1:41721)
0806 10:47:13.885262 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.885266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.886489 (+ 1223us) server_negotiation.cc:407] Connection header received
0806 10:47:13.886593 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.886599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.886675 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.886768 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:13.887270 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.888019 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.889759 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.890576 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.890726 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.890815 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.891273 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.891416 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:47:13.891517 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91833ff577074e2bb0201e516f78e41b" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f74fa4300748f58f4575217f13cd42" is_pre_election: true
I20260806 10:47:13.891773 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.884959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:41721 (local address 127.16.21.3:53847)
0806 10:47:13.886374 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:13.886388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:13.886485 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:13.886788 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:13.886791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:13.886809 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:13.887085 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.887092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.888173 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.888184 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.889573 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:13.889585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.890904 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.890909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:13.891109 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.891134 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 10:47:13.891245 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:47:13.891381 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1318,"mutex_wait_us":474,"thread_start_us":568,"threads_started":1}
I20260806 10:47:13.892095 17921 raft_consensus.cc:2454] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 0.
I20260806 10:47:13.892796 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91833ff577074e2bb0201e516f78e41b" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be832b12afd46fb8c15e2036e30d841" is_pre_election: true
I20260806 10:47:13.892880 17989 leader_election.cc:302] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1f74fa4300748f58f4575217f13cd42, d6fbdda51a5f42f38ebeb796d2c693f3; no voters:
I20260806 10:47:13.893335 18138 raft_consensus.cc:2781] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:13.893365 17775 raft_consensus.cc:2454] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 0.
I20260806 10:47:13.893501 18138 raft_consensus.cc:482] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:13.893606 18138 raft_consensus.cc:3037] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:13.895892 18138 raft_consensus.cc:504] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.896579 18138 leader_election.cc:288] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 election: Requested vote from peers c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041), 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721)
I20260806 10:47:13.897014 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91833ff577074e2bb0201e516f78e41b" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f74fa4300748f58f4575217f13cd42"
I20260806 10:47:13.897038 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91833ff577074e2bb0201e516f78e41b" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be832b12afd46fb8c15e2036e30d841"
I20260806 10:47:13.897220 17921 raft_consensus.cc:3037] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:13.897267 17775 raft_consensus.cc:3037] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:13.899871 17921 raft_consensus.cc:2454] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 1.
I20260806 10:47:13.900000 17775 raft_consensus.cc:2454] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 1.
I20260806 10:47:13.900280 17989 leader_election.cc:302] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1f74fa4300748f58f4575217f13cd42, d6fbdda51a5f42f38ebeb796d2c693f3; no voters:
I20260806 10:47:13.900578 18138 raft_consensus.cc:2781] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:13.901041 18138 raft_consensus.cc:686] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 LEADER]: Becoming Leader. State: Replica: d6fbdda51a5f42f38ebeb796d2c693f3, State: Running, Role: LEADER
I20260806 10:47:13.901573 18138 consensus_queue.cc:237] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } }
I20260806 10:47:13.906179 17613 catalog_manager.cc:5495] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } health_report { overall_health: UNKNOWN } } }
I20260806 10:47:13.973508 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:13.968059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.16.21.3:41201)
0806 10:47:13.968308 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:13.968315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:13.968539 (+ 224us) server_negotiation.cc:407] Connection header received
0806 10:47:13.968785 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:13.968789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:13.968865 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:13.968977 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:13.969288 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.970015 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.971268 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:13.971659 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:13.971748 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:13.972150 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:13.972802 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:13.972903 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:13.973023 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 10:47:13.973082 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:47:13.980868 17921 raft_consensus.cc:1261] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Refusing update from remote peer d6fbdda51a5f42f38ebeb796d2c693f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:13.981097 17775 raft_consensus.cc:1261] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Refusing update from remote peer d6fbdda51a5f42f38ebeb796d2c693f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:13.981835 18140 consensus_queue.cc:1035] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:13.982187 18138 consensus_queue.cc:1035] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:13.994381 18143 mvcc.cc:205] Tried to move back new op lower bound from 7315510206376615936 to 7315510206065000448. Current Snapshot: MvccSnapshot[applied={T|T < 7315510206376615936}]
I20260806 10:47:14.011865 17613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:14.012190 17613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:14.015942 17613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: Sending DeleteTablet for 3 replicas of tablet 91833ff577074e2bb0201e516f78e41b
I20260806 10:47:14.017102 17897 tablet_service.cc:1514] Processing DeleteTablet for tablet 91833ff577074e2bb0201e516f78e41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:14 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:58886
I20260806 10:47:14.017261 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet 91833ff577074e2bb0201e516f78e41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:14 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:38646
I20260806 10:47:14.017230 17754 tablet_service.cc:1514] Processing DeleteTablet for tablet 91833ff577074e2bb0201e516f78e41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:14 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260806 10:47:14.021601 17613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:14.023272 17613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:14.027554 18154 tablet_replica.cc:330] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: stopping tablet replica
I20260806 10:47:14.028040 18154 raft_consensus.cc:2229] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:14.028232 18153 tablet_replica.cc:330] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: stopping tablet replica
I20260806 10:47:14.028578 18154 raft_consensus.cc:2258] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:14.028710 18153 raft_consensus.cc:2229] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:14.029249 18153 raft_consensus.cc:2258] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:14.031736 18153 ts_tablet_manager.cc:1905] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:14.036487 17754 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1b59d067c240e08a466f2e7b339138 (DEFAULT_TABLE table=test-table [id=7a2e32b42312450e8f50c729fd4bf914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:14.036667 18042 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1b59d067c240e08a466f2e7b339138 (DEFAULT_TABLE table=test-table [id=7a2e32b42312450e8f50c729fd4bf914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:14.036950 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1b59d067c240e08a466f2e7b339138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:14.037281 18042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1b59d067c240e08a466f2e7b339138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:14.040723 17896 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1b59d067c240e08a466f2e7b339138 (DEFAULT_TABLE table=test-table [id=7a2e32b42312450e8f50c729fd4bf914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:14.041209 17896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1b59d067c240e08a466f2e7b339138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:14.044376 18131 tablet_bootstrap.cc:492] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Bootstrap starting.
I20260806 10:47:14.046715 18131 tablet_bootstrap.cc:654] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:14.047842 18154 ts_tablet_manager.cc:1905] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:14.049081 18130 tablet_bootstrap.cc:492] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Bootstrap starting.
I20260806 10:47:14.050833 18130 tablet_bootstrap.cc:654] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:14.052187 18129 tablet_bootstrap.cc:492] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Bootstrap starting.
I20260806 10:47:14.054242 18129 tablet_bootstrap.cc:654] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:14.063372 18153 ts_tablet_manager.cc:1918] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:14.064234 18155 tablet_replica.cc:330] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: stopping tablet replica
I20260806 10:47:14.065106 18155 raft_consensus.cc:2229] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:14.065948 18153 log.cc:1196] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/91833ff577074e2bb0201e516f78e41b
I20260806 10:47:14.066566 18153 ts_tablet_manager.cc:1939] T 91833ff577074e2bb0201e516f78e41b P c1f74fa4300748f58f4575217f13cd42: Deleting consensus metadata
I20260806 10:47:14.066987 18155 raft_consensus.cc:2258] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:14.068750 18131 tablet_bootstrap.cc:492] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: No bootstrap required, opened a new log
I20260806 10:47:14.068917 18131 ts_tablet_manager.cc:1397] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.005s
I20260806 10:47:14.068986 18129 tablet_bootstrap.cc:492] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: No bootstrap required, opened a new log
I20260806 10:47:14.069093 18130 tablet_bootstrap.cc:492] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: No bootstrap required, opened a new log
I20260806 10:47:14.069147 18129 ts_tablet_manager.cc:1397] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20260806 10:47:14.069331 18130 ts_tablet_manager.cc:1397] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.002s
I20260806 10:47:14.069628 18131 raft_consensus.cc:348] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.069890 18131 raft_consensus.cc:374] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:14.070003 18131 raft_consensus.cc:729] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6fbdda51a5f42f38ebeb796d2c693f3, State: Initialized, Role: FOLLOWER
I20260806 10:47:14.069934 18129 raft_consensus.cc:348] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.070091 18154 ts_tablet_manager.cc:1918] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:14.070168 18129 raft_consensus.cc:374] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:14.070250 18154 log.cc:1196] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/91833ff577074e2bb0201e516f78e41b
I20260806 10:47:14.070286 18129 raft_consensus.cc:729] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f74fa4300748f58f4575217f13cd42, State: Initialized, Role: FOLLOWER
I20260806 10:47:14.070616 18129 consensus_queue.cc:260] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.070834 18154 ts_tablet_manager.cc:1939] T 91833ff577074e2bb0201e516f78e41b P 0be832b12afd46fb8c15e2036e30d841: Deleting consensus metadata
I20260806 10:47:14.070621 18130 raft_consensus.cc:348] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.071034 18130 raft_consensus.cc:374] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:14.071200 18130 raft_consensus.cc:729] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be832b12afd46fb8c15e2036e30d841, State: Initialized, Role: FOLLOWER
I20260806 10:47:14.071517 18155 ts_tablet_manager.cc:1905] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:14.071583 18130 consensus_queue.cc:260] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.073443 17598 catalog_manager.cc:4841] TS c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041): tablet 91833ff577074e2bb0201e516f78e41b (table test-table [id=6b1a458f00ff41f5acf1a2569aef6f18]) successfully deleted
I20260806 10:47:14.073400 18131 consensus_queue.cc:260] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.073990 18131 ts_tablet_manager.cc:1428] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 10:47:14.074223 18129 ts_tablet_manager.cc:1428] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260806 10:47:14.074488 18148 raft_consensus.cc:482] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:14.074621 18148 raft_consensus.cc:504] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.075341 18148 leader_election.cc:288] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721), c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041)
I20260806 10:47:14.075335 17599 catalog_manager.cc:4841] TS 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721): tablet 91833ff577074e2bb0201e516f78e41b (table test-table [id=6b1a458f00ff41f5acf1a2569aef6f18]) successfully deleted
I20260806 10:47:14.075982 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1b59d067c240e08a466f2e7b339138" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f74fa4300748f58f4575217f13cd42" is_pre_election: true
I20260806 10:47:14.076192 17921 raft_consensus.cc:2454] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 0.
I20260806 10:47:14.076236 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1b59d067c240e08a466f2e7b339138" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be832b12afd46fb8c15e2036e30d841" is_pre_election: true
I20260806 10:47:14.076512 17775 raft_consensus.cc:2454] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 0.
I20260806 10:47:14.076642 17989 leader_election.cc:302] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1f74fa4300748f58f4575217f13cd42, d6fbdda51a5f42f38ebeb796d2c693f3; no voters:
I20260806 10:47:14.076921 18148 raft_consensus.cc:2781] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:14.077037 18148 raft_consensus.cc:482] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:14.077127 18148 raft_consensus.cc:3037] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:14.077999 18130 ts_tablet_manager.cc:1428] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260806 10:47:14.080159 18148 raft_consensus.cc:504] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.081149 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1b59d067c240e08a466f2e7b339138" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be832b12afd46fb8c15e2036e30d841"
I20260806 10:47:14.081332 17775 raft_consensus.cc:3037] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:14.081380 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1b59d067c240e08a466f2e7b339138" candidate_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f74fa4300748f58f4575217f13cd42"
I20260806 10:47:14.081597 17921 raft_consensus.cc:3037] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:14.084116 17775 raft_consensus.cc:2454] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 1.
I20260806 10:47:14.084553 17990 leader_election.cc:302] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be832b12afd46fb8c15e2036e30d841, d6fbdda51a5f42f38ebeb796d2c693f3; no voters:
I20260806 10:47:14.084601 17921 raft_consensus.cc:2454] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6fbdda51a5f42f38ebeb796d2c693f3 in term 1.
I20260806 10:47:14.084926 18138 raft_consensus.cc:2781] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:14.085111 18138 raft_consensus.cc:686] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 LEADER]: Becoming Leader. State: Replica: d6fbdda51a5f42f38ebeb796d2c693f3, State: Running, Role: LEADER
I20260806 10:47:14.085428 18138 consensus_queue.cc:237] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.085110 18148 leader_election.cc:288] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [CANDIDATE]: Term 1 election: Requested vote from peers 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721), c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041)
I20260806 10:47:14.091684 18155 ts_tablet_manager.cc:1918] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:14.092972 18155 log.cc:1196] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/91833ff577074e2bb0201e516f78e41b
I20260806 10:47:14.093578 18155 ts_tablet_manager.cc:1939] T 91833ff577074e2bb0201e516f78e41b P d6fbdda51a5f42f38ebeb796d2c693f3: Deleting consensus metadata
I20260806 10:47:14.093429 17611 catalog_manager.cc:5495] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } health_report { overall_health: HEALTHY } } }
I20260806 10:47:14.094978 17599 catalog_manager.cc:4841] TS d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3:41201): tablet 91833ff577074e2bb0201e516f78e41b (table test-table [id=6b1a458f00ff41f5acf1a2569aef6f18]) successfully deleted
Aug 06 10:47:14 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013231, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:14.144866 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.130762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.16.21.62:35895)
0806 10:47:14.131010 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.131019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.131624 (+ 605us) server_negotiation.cc:407] Connection header received
0806 10:47:14.131846 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.131854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.131919 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.132022 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:14.133272 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.134003 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.135497 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.135970 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.136111 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.139559 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:14.139563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:14.139566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:14.139593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:14.142165 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.142850 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.142861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.142864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.142969 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.143336 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.143342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.143345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.143620 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:14.143777 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.144196 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.144554 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":76}
I20260806 10:47:14.144905 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.130740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35895 (local address 127.0.0.1:41364)
0806 10:47:14.131517 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:14.131539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:14.131714 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:14.132051 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:14.132055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:14.132848 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:14.133121 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.133131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.134140 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.134148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.135358 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.135371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.136149 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.136155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.136346 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.137050 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:14.137067 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:14.139382 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:14.142290 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:14.142305 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:14.142314 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:14.142716 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:14.143085 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:14.143091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:14.143094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:14.143229 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:14.143751 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:14.143758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:14.143942 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 10:47:14.144112 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 10:47:14.144420 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":475,"threads_started":1}
I20260806 10:47:14.150214 17611 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41364:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 10:47:14.151528 17611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:14.158739 17754 tablet_service.cc:1467] Processing CreateTablet for tablet f8eecb32d60f4c76a1c3186499ecf236 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0efefbe5145946de833b7af407519791]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:47:14.159026 18043 tablet_service.cc:1467] Processing CreateTablet for tablet f8eecb32d60f4c76a1c3186499ecf236 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0efefbe5145946de833b7af407519791]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:47:14.159368 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8eecb32d60f4c76a1c3186499ecf236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:14.159231 17896 tablet_service.cc:1467] Processing CreateTablet for tablet f8eecb32d60f4c76a1c3186499ecf236 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0efefbe5145946de833b7af407519791]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:47:14.159641 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8eecb32d60f4c76a1c3186499ecf236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:14.159755 17896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8eecb32d60f4c76a1c3186499ecf236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:14.166932 18130 tablet_bootstrap.cc:492] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: Bootstrap starting.
I20260806 10:47:14.167284 18131 tablet_bootstrap.cc:492] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: Bootstrap starting.
I20260806 10:47:14.168421 18130 tablet_bootstrap.cc:654] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:14.168884 18129 tablet_bootstrap.cc:492] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: Bootstrap starting.
I20260806 10:47:14.169047 18131 tablet_bootstrap.cc:654] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:14.170915 18129 tablet_bootstrap.cc:654] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:14.179479 18129 tablet_bootstrap.cc:492] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: No bootstrap required, opened a new log
I20260806 10:47:14.179706 18129 ts_tablet_manager.cc:1397] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260806 10:47:14.180140 18130 tablet_bootstrap.cc:492] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: No bootstrap required, opened a new log
I20260806 10:47:14.180356 18130 ts_tablet_manager.cc:1397] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260806 10:47:14.180509 18131 tablet_bootstrap.cc:492] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: No bootstrap required, opened a new log
I20260806 10:47:14.180537 18129 raft_consensus.cc:348] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.180732 18131 ts_tablet_manager.cc:1397] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260806 10:47:14.180802 18129 raft_consensus.cc:374] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:14.180878 18129 raft_consensus.cc:729] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f74fa4300748f58f4575217f13cd42, State: Initialized, Role: FOLLOWER
I20260806 10:47:14.181066 18130 raft_consensus.cc:348] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.181304 18130 raft_consensus.cc:374] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:14.181195 18129 consensus_queue.cc:260] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.181381 18130 raft_consensus.cc:729] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be832b12afd46fb8c15e2036e30d841, State: Initialized, Role: FOLLOWER
I20260806 10:47:14.181684 18130 consensus_queue.cc:260] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.181736 18137 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.182142 18137 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.182583 18129 ts_tablet_manager.cc:1428] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:14.184082 18131 raft_consensus.cc:348] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.184455 18131 raft_consensus.cc:374] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:14.184612 18131 raft_consensus.cc:729] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6fbdda51a5f42f38ebeb796d2c693f3, State: Initialized, Role: FOLLOWER
I20260806 10:47:14.184922 18131 consensus_queue.cc:260] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.185449 18138 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.185774 18138 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.182107 18136 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.186203 18136 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.186655 18130 ts_tablet_manager.cc:1428] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260806 10:47:14.188450 18131 ts_tablet_manager.cc:1428] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 10:47:14.447060 18137 raft_consensus.cc:482] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:14.447256 18137 raft_consensus.cc:504] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.448982 18137 leader_election.cc:288] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721), d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3:41201)
I20260806 10:47:14.456985 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.449562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:41355 (local address 127.16.21.1:41721)
0806 10:47:14.450482 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.450491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.450515 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:47:14.450608 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.450616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.450724 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.451534 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:14.451724 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.452793 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.455075 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.455983 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.456117 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.456215 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.456540 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.456672 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":1045,"threads_started":1}
I20260806 10:47:14.457131 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.449141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:41721 (local address 127.16.21.2:41355)
0806 10:47:14.449835 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:14.449855 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:14.450013 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:14.450919 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:14.450928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:14.450973 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:14.451348 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.451363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.452948 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.452953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.454913 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.454925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.456131 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.456137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.456363 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.456387 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:47:14.456514 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 10:47:14.456698 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"thread_start_us":486,"threads_started":1}
I20260806 10:47:14.457886 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8eecb32d60f4c76a1c3186499ecf236" candidate_uuid: "c1f74fa4300748f58f4575217f13cd42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be832b12afd46fb8c15e2036e30d841" is_pre_election: true
I20260806 10:47:14.458158 17775 raft_consensus.cc:2454] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1f74fa4300748f58f4575217f13cd42 in term 0.
I20260806 10:47:14.458693 17849 leader_election.cc:302] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be832b12afd46fb8c15e2036e30d841, c1f74fa4300748f58f4575217f13cd42; no voters:
I20260806 10:47:14.459182 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.450408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:54107 (local address 127.16.21.3:41201)
0806 10:47:14.450594 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.450600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.452146 (+ 1546us) server_negotiation.cc:407] Connection header received
0806 10:47:14.452358 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.452364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.452441 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.452530 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:14.453215 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.454405 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.456922 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.458124 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.458267 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.458360 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.458743 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.458884 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:47:14.459188 18137 raft_consensus.cc:2781] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:14.459329 18137 raft_consensus.cc:482] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:14.459381 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.450050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41201 (local address 127.16.21.2:54107)
0806 10:47:14.452022 (+ 1972us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:14.452039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:14.452180 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:14.452571 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:14.452576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:14.452601 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:14.453006 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.453022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.454551 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.454557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.456729 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.456744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.458360 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.458367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.458566 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.458605 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:47:14.458716 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:47:14.458857 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"mutex_wait_us":24,"thread_start_us":1630,"threads_started":1}
I20260806 10:47:14.459442 18137 raft_consensus.cc:3037] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:14.460469 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8eecb32d60f4c76a1c3186499ecf236" candidate_uuid: "c1f74fa4300748f58f4575217f13cd42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" is_pre_election: true
I20260806 10:47:14.460884 18063 raft_consensus.cc:2454] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1f74fa4300748f58f4575217f13cd42 in term 0.
I20260806 10:47:14.462595 18137 raft_consensus.cc:504] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.463218 18137 leader_election.cc:288] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [CANDIDATE]: Term 1 election: Requested vote from peers 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721), d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3:41201)
I20260806 10:47:14.463588 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8eecb32d60f4c76a1c3186499ecf236" candidate_uuid: "c1f74fa4300748f58f4575217f13cd42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be832b12afd46fb8c15e2036e30d841"
I20260806 10:47:14.463830 17775 raft_consensus.cc:3037] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:14.463830 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8eecb32d60f4c76a1c3186499ecf236" candidate_uuid: "c1f74fa4300748f58f4575217f13cd42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3"
I20260806 10:47:14.464020 18063 raft_consensus.cc:3037] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:14.467010 17775 raft_consensus.cc:2454] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1f74fa4300748f58f4575217f13cd42 in term 1.
I20260806 10:47:14.467422 17849 leader_election.cc:302] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be832b12afd46fb8c15e2036e30d841, c1f74fa4300748f58f4575217f13cd42; no voters:
I20260806 10:47:14.467430 18063 raft_consensus.cc:2454] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1f74fa4300748f58f4575217f13cd42 in term 1.
I20260806 10:47:14.468362 18137 raft_consensus.cc:2781] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:14.468912 18137 raft_consensus.cc:686] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [term 1 LEADER]: Becoming Leader. State: Replica: c1f74fa4300748f58f4575217f13cd42, State: Running, Role: LEADER
I20260806 10:47:14.469316 18137 consensus_queue.cc:237] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } }
I20260806 10:47:14.471653 18172 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: TxnStatusTablet state changed. Reason: New leader c1f74fa4300748f58f4575217f13cd42. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.472052 18172 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:47:14.472922 18174 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 10:47:14.473278 18174 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 10:47:14.473183 17611 catalog_manager.cc:5495] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1f74fa4300748f58f4575217f13cd42 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } health_report { overall_health: UNKNOWN } } }
I20260806 10:47:14.519425 18138 consensus_queue.cc:1035] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:14.527009 18140 consensus_queue.cc:1035] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:14.558358 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.539302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:36041 (local address 127.0.0.1:58892)
0806 10:47:14.539769 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:14.539803 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:14.539950 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:14.551493 (+ 11543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:14.551497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:14.551513 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:14.551739 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.551751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.553346 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.553351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.554985 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.555001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.556148 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.556154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.556382 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.556457 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:14.557731 (+ 1274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:14.557761 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:47:14.557832 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 10:47:14.557988 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
I20260806 10:47:14.559285 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.539362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.16.21.2:36041)
0806 10:47:14.551132 (+ 11770us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.551136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.551155 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:47:14.551221 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.551225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.551303 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.551469 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:14.551931 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.553185 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.555179 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.555951 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.556633 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.556760 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:14.557569 (+ 809us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:14.558756 (+ 1187us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.558843 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.558934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11653,"thread_start_us":11514,"threads_started":1}
I20260806 10:47:14.563767 17775 raft_consensus.cc:1261] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Refusing update from remote peer c1f74fa4300748f58f4575217f13cd42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:14.563899 18063 raft_consensus.cc:1261] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Refusing update from remote peer c1f74fa4300748f58f4575217f13cd42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:14.564458 18137 consensus_queue.cc:1035] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:14.564766 18172 consensus_queue.cc:1035] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:14.569092 18136 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: TxnStatusTablet state changed. Reason: New leader c1f74fa4300748f58f4575217f13cd42. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.569401 18136 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.571573 18178 mvcc.cc:205] Tried to move back new op lower bound from 7315510208766488576 to 7315510208389844992. Current Snapshot: MvccSnapshot[applied={T|T < 7315510208766488576}]
I20260806 10:47:14.571702 18140 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: TxnStatusTablet state changed. Reason: New leader c1f74fa4300748f58f4575217f13cd42. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.572072 18140 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.572605 18182 mvcc.cc:205] Tried to move back new op lower bound from 7315510208766488576 to 7315510208389844992. Current Snapshot: MvccSnapshot[applied={T|T < 7315510208766488576}]
I20260806 10:47:14.581410 18136 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.581629 18136 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P 0be832b12afd46fb8c15e2036e30d841: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.585415 18172 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.585718 18172 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:47:14.606871 18137 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.607327 18137 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P c1f74fa4300748f58f4575217f13cd42: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:47:14.621129 18140 tablet_replica.cc:439] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1f74fa4300748f58f4575217f13cd42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be832b12afd46fb8c15e2036e30d841" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 41721 } } peers { permanent_uuid: "c1f74fa4300748f58f4575217f13cd42" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 36041 } } peers { permanent_uuid: "d6fbdda51a5f42f38ebeb796d2c693f3" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41201 } } }
I20260806 10:47:14.621392 18140 tablet_replica.cc:442] T f8eecb32d60f4c76a1c3186499ecf236 P d6fbdda51a5f42f38ebeb796d2c693f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:14.631861 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.626043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:42969 (local address 127.16.21.2:36041)
0806 10:47:14.626242 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.626250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.626803 (+ 553us) server_negotiation.cc:407] Connection header received
0806 10:47:14.627012 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.627018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.627092 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.627189 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:14.627741 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.628634 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.630095 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.630511 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.630636 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.631104 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:14.631200 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:14.631315 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.631451 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.631524 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 10:47:14.632195 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.626048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:36041 (local address 127.16.21.3:42969)
0806 10:47:14.626700 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:14.626716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:14.626880 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:14.627209 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:14.627214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:14.627226 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:14.627519 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.627527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.628774 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.628779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.629933 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.629944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.630702 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.630709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.630861 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.630909 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:14.631317 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:14.631330 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 10:47:14.631427 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 10:47:14.631552 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":370,"threads_started":1}
I20260806 10:47:14.672386 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.664997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:55907 (local address 127.16.21.3:41201)
0806 10:47:14.665175 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.665183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.665861 (+ 678us) server_negotiation.cc:407] Connection header received
0806 10:47:14.666097 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.666102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.666215 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.666319 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:14.667170 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.668040 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.670365 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.670888 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.670990 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.671571 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:14.671702 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:14.671796 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.671971 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.672065 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:47:14.672751 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.665006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41201 (local address 127.16.21.2:55907)
0806 10:47:14.665761 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:14.665778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:14.665958 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:14.666394 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:14.666401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:14.666421 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:14.666913 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.666932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.668171 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.668176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.670214 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:14.670229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.671089 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.671096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:14.671283 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.671352 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:14.671825 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:14.671836 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 10:47:14.671951 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 10:47:14.672096 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":375,"threads_started":1}
I20260806 10:47:14.705498 17612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:14.705725 17612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:14.708797 17612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4744c44d430b4d6e9cd6931017e5af83: Sending DeleteTablet for 3 replicas of tablet 6f1b59d067c240e08a466f2e7b339138
I20260806 10:47:14.709533 17754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1b59d067c240e08a466f2e7b339138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:14 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260806 10:47:14.709586 17896 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1b59d067c240e08a466f2e7b339138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:14 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:58886
I20260806 10:47:14.710213 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1b59d067c240e08a466f2e7b339138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:14 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:38646
I20260806 10:47:14.718832 18196 tablet_replica.cc:330] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: stopping tablet replica
I20260806 10:47:14.719101 18196 raft_consensus.cc:2229] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:14.719403 18196 raft_consensus.cc:2258] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:14.721627 18196 ts_tablet_manager.cc:1905] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:14.724860 18198 tablet_replica.cc:330] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: stopping tablet replica
I20260806 10:47:14.725162 18198 raft_consensus.cc:2229] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:14.725629 18198 raft_consensus.cc:2258] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:14.726759 18198 ts_tablet_manager.cc:1905] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:14.728456 18197 tablet_replica.cc:330] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: stopping tablet replica
I20260806 10:47:14.728674 18197 raft_consensus.cc:2229] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:14.728976 18197 raft_consensus.cc:2258] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:14.729729 18196 ts_tablet_manager.cc:1918] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:14.729866 18196 log.cc:1196] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/6f1b59d067c240e08a466f2e7b339138
I20260806 10:47:14.730178 18197 ts_tablet_manager.cc:1905] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:14.730360 18196 ts_tablet_manager.cc:1939] T 6f1b59d067c240e08a466f2e7b339138 P 0be832b12afd46fb8c15e2036e30d841: Deleting consensus metadata
I20260806 10:47:14.731448 17599 catalog_manager.cc:4841] TS 0be832b12afd46fb8c15e2036e30d841 (127.16.21.1:41721): tablet 6f1b59d067c240e08a466f2e7b339138 (table test-table [id=7a2e32b42312450e8f50c729fd4bf914]) successfully deleted
I20260806 10:47:14.735035 18198 ts_tablet_manager.cc:1918] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:14.735272 18198 log.cc:1196] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/6f1b59d067c240e08a466f2e7b339138
I20260806 10:47:14.735772 18198 ts_tablet_manager.cc:1939] T 6f1b59d067c240e08a466f2e7b339138 P d6fbdda51a5f42f38ebeb796d2c693f3: Deleting consensus metadata
I20260806 10:47:14.736901 17599 catalog_manager.cc:4841] TS d6fbdda51a5f42f38ebeb796d2c693f3 (127.16.21.3:41201): tablet 6f1b59d067c240e08a466f2e7b339138 (table test-table [id=7a2e32b42312450e8f50c729fd4bf914]) successfully deleted
I20260806 10:47:14.737895 18197 ts_tablet_manager.cc:1918] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:14.738061 18197 log.cc:1196] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/6f1b59d067c240e08a466f2e7b339138
I20260806 10:47:14.738622 18197 ts_tablet_manager.cc:1939] T 6f1b59d067c240e08a466f2e7b339138 P c1f74fa4300748f58f4575217f13cd42: Deleting consensus metadata
Aug 06 10:47:14 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:14.739724 17598 catalog_manager.cc:4841] TS c1f74fa4300748f58f4575217f13cd42 (127.16.21.2:36041): tablet 6f1b59d067c240e08a466f2e7b339138 (table test-table [id=7a2e32b42312450e8f50c729fd4bf914]) successfully deleted
I20260806 10:47:14.746222 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.720963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.16.21.1:41721)
0806 10:47:14.721150 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.721160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.724339 (+ 3179us) server_negotiation.cc:407] Connection header received
0806 10:47:14.725361 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.725367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.725471 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.725538 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:14.731320 (+ 5782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.732409 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.734339 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.734968 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.735063 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.740433 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:14.740447 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:14.740471 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:14.740500 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:14.743429 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.744121 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.744128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.744130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.744325 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.744716 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.744721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.744724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.745067 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:14.745386 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.745678 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.745916 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 10:47:14.746871 17821 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50532
I20260806 10:47:14.763487 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.749682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.16.21.62:35895)
0806 10:47:14.750417 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.750423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.750442 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:47:14.750497 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.750502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.750583 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.750688 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:14.751937 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.753137 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.754338 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.755038 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.755165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.757847 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:14.757853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:14.757858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:14.757916 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:14.761103 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.761683 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.761688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.761691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.761844 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.762145 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.762149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.762152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.762469 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:14.762709 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.762922 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.763123 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":447,"threads_started":1}
W20260806 10:47:14.764091 17612 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41380
I20260806 10:47:14.765837 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17686
I20260806 10:47:14.779960 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.767664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.16.21.1:41721)
0806 10:47:14.767991 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.767997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.768201 (+ 204us) server_negotiation.cc:407] Connection header received
0806 10:47:14.768437 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.768442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.768513 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.768612 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:14.769674 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.770371 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.771566 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.771983 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.772122 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.774434 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:14.774440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:14.774445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:14.774483 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:14.777244 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.778061 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.778067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.778070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.778199 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.778584 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.778587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.778589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.778847 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:14.779091 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.779403 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.779598 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
W20260806 10:47:14.780609 17821 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50542
W20260806 10:47:14.781147 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17686: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:14.781337 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17686
I20260806 10:47:14.803097 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17830
Aug 06 10:47:14 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:14.820187 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.803910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.16.21.2:36041)
0806 10:47:14.804133 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.804139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.804160 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:47:14.804236 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.804241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.804368 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.804476 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:14.805697 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.806954 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.808296 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.809034 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.809193 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.813363 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:14.813375 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:14.813391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:14.813442 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:14.817436 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.818029 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.818043 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.818049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.818207 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.818561 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.818566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.818569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.819004 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:14.819371 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.819559 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.819783 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 10:47:14.820983 17963 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58908
W20260806 10:47:14.821656 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:14.821820 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17830
I20260806 10:47:14.845531 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17972
Aug 06 10:47:14 dist-test-slave-0b58 krb5kdc[17568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:14.859572 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:14.846208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.16.21.3:41201)
0806 10:47:14.846376 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:14.846380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:14.846398 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:47:14.846479 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:14.846483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:14.846544 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:14.846633 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:14.847794 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.848603 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.849798 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:14.850423 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:14.850522 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:14.854543 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:14.854558 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:14.854573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:14.854604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:14.857329 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.857907 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.857914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.857916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.858053 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:14.858400 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:14.858403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:14.858405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:14.858729 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:14.858970 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:14.859159 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 10:47:14.859323 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 10:47:14.860244 18105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38668
W20260806 10:47:14.860929 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17972: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:14.861092 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17972
I20260806 10:47:14.886541 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 17581
I20260806 10:47:14.887719 17655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:15.006881 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 17581
2026-08-06T10:47:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4796 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:15 dist-test-slave-0b58 krb5kdc[18208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:15 dist-test-slave-0b58 krb5kdc[18208](info): set up 2 sockets
Aug 06 10:47:15 dist-test-slave-0b58 krb5kdc[18208](info): commencing operation
krb5kdc: starting...
W20260806 10:47:16.124338 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:16 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:16.333617 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:39517
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:39517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:16.656481 18222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:16.656770 18222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:16.656924 18222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:16.664036 18222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:16.664129 18222 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 10:47:16.664222 18222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:16.664356 18222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:16.664462 18222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:16.664570 18222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:16.674295 18222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:39517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:16.675498 18222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:16.676815 18222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:16.688795 18228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:16.688854 18230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:16.688828 18227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:16.689832 18222 server_base.cc:1034] running on GCE node
I20260806 10:47:16.690536 18222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:16.691869 18222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:16.693110 18222 hybrid_clock.cc:648] HybridClock initialized: now 1786013236693021 us; error 47 us; skew 500 ppm
I20260806 10:47:16.693464 18222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:16 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013236, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:16.696593 18222 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:16.698019 18222 webserver.cc:460] Webserver started at http://127.16.21.62:36981/ using document root <none> and password file <none>
I20260806 10:47:16.698709 18222 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:16.698841 18222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:16.699158 18222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:16.702086 18222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "4b2e684469bb4d1a832783b994d4b35b"
format_stamp: "Formatted at 2026-08-06 10:47:16 on dist-test-slave-0b58"
server_key: "83eb90ced94a16214b0a7e2945c534e4"
server_key_iv: "1b8d3d3eae1474aa520e1edc77380242"
server_key_version: "encryptionkey@0"
I20260806 10:47:16.702862 18222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "4b2e684469bb4d1a832783b994d4b35b"
format_stamp: "Formatted at 2026-08-06 10:47:16 on dist-test-slave-0b58"
server_key: "83eb90ced94a16214b0a7e2945c534e4"
server_key_iv: "1b8d3d3eae1474aa520e1edc77380242"
server_key_version: "encryptionkey@0"
I20260806 10:47:16.707825 18222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:47:16.711172 18237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:16.712549 18222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:16.712734 18222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "4b2e684469bb4d1a832783b994d4b35b"
format_stamp: "Formatted at 2026-08-06 10:47:16 on dist-test-slave-0b58"
server_key: "83eb90ced94a16214b0a7e2945c534e4"
server_key_iv: "1b8d3d3eae1474aa520e1edc77380242"
server_key_version: "encryptionkey@0"
I20260806 10:47:16.712989 18222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:16.746236 18222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:16.750195 18222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:16.750479 18222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:16.773307 18222 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:39517
I20260806 10:47:16.773341 18300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:39517 every 8 connection(s)
I20260806 10:47:16.774631 18222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:16.778306 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:16.781176 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 18222
I20260806 10:47:16.781388 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:16.782039 16468 external_mini_cluster.cc:1428] Setting key a9c1bae4f3603c0b612054036fef1ece
I20260806 10:47:16.791224 18301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Bootstrap starting.
I20260806 10:47:16.795024 18301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:16.796412 18301 log.cc:824] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:47:16 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:16.799190 18301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: No bootstrap required, opened a new log
I20260806 10:47:16.806277 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:16.783960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.16.21.62:39517)
0806 10:47:16.784800 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:16.784815 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:16.784864 (+ 49us) server_negotiation.cc:407] Connection header received
0806 10:47:16.787007 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:16.787038 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:16.787517 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:16.787968 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:16.789309 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:16.791358 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:16.793250 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:16.793905 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:16.794034 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:16.798911 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:16.798919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:16.798932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:16.798972 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:16.802213 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:16.802951 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:16.802957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:16.802963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:16.803130 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:16.803545 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:16.803549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:16.803551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:16.804064 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:16.804408 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:16.804890 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 10:47:16.805286 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":379,"threads_started":1}
I20260806 10:47:16.807730 18301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e684469bb4d1a832783b994d4b35b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 39517 } }
I20260806 10:47:16.808117 18301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:16.808248 18301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2e684469bb4d1a832783b994d4b35b, State: Initialized, Role: FOLLOWER
I20260806 10:47:16.808874 18301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e684469bb4d1a832783b994d4b35b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 39517 } }
I20260806 10:47:16.809160 18301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:16.809343 18301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:16.809544 18301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:16.811743 18301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e684469bb4d1a832783b994d4b35b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 39517 } }
I20260806 10:47:16.812238 18301 leader_election.cc:302] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2e684469bb4d1a832783b994d4b35b; no voters:
I20260806 10:47:16.812909 18301 leader_election.cc:288] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:16.813020 18306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:16.814396 18306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [term 1 LEADER]: Becoming Leader. State: Replica: 4b2e684469bb4d1a832783b994d4b35b, State: Running, Role: LEADER
I20260806 10:47:16.814931 18306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e684469bb4d1a832783b994d4b35b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 39517 } }
I20260806 10:47:16.815155 18301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:16.818946 18308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2e684469bb4d1a832783b994d4b35b. Latest consensus state: current_term: 1 leader_uuid: "4b2e684469bb4d1a832783b994d4b35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e684469bb4d1a832783b994d4b35b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 39517 } } }
I20260806 10:47:16.818929 18307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2e684469bb4d1a832783b994d4b35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e684469bb4d1a832783b994d4b35b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 39517 } } }
I20260806 10:47:16.819495 18308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:16.819563 18307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:16.820716 18315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:16.825544 18315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:16.833873 18315 catalog_manager.cc:1347] Generated new cluster ID: d99f8dd84f984357b56dde4c4f9978c2
I20260806 10:47:16.833979 18315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:16.855860 18315 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:16.857683 18315 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:16.866680 18315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Generated new TSK 0
I20260806 10:47:16.867651 18315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:16.918174 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:39517
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:17.248044 18327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:17.248320 18327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:17.248422 18327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:17.255620 18327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:17.255721 18327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:17.255811 18327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 10:47:17.255863 18327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:47:17.255932 18327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:17.265630 18327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:17.266793 18327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:17.267995 18327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:17.280989 18332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:17.281013 18333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:17.281008 18335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:17.281774 18327 server_base.cc:1034] running on GCE node
I20260806 10:47:17.282344 18327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:17.283218 18327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:17.284442 18327 hybrid_clock.cc:648] HybridClock initialized: now 1786013237284389 us; error 50 us; skew 500 ppm
I20260806 10:47:17.284771 18327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:17 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013237, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:17.287873 18327 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:17.289322 18327 webserver.cc:460] Webserver started at http://127.16.21.1:46209/ using document root <none> and password file <none>
I20260806 10:47:17.289945 18327 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:17.290086 18327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:17.290390 18327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:17.293264 18327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "a5d84185dda64117b3d793e359da9dd8"
format_stamp: "Formatted at 2026-08-06 10:47:17 on dist-test-slave-0b58"
server_key: "92a1edfe1e9cdf19e53ae729823bacc3"
server_key_iv: "d35cb62093c568d03a9f27b872e187f7"
server_key_version: "encryptionkey@0"
I20260806 10:47:17.293994 18327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "a5d84185dda64117b3d793e359da9dd8"
format_stamp: "Formatted at 2026-08-06 10:47:17 on dist-test-slave-0b58"
server_key: "92a1edfe1e9cdf19e53ae729823bacc3"
server_key_iv: "d35cb62093c568d03a9f27b872e187f7"
server_key_version: "encryptionkey@0"
I20260806 10:47:17.298835 18327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:47:17.302186 18342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:17.303411 18327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:17.303582 18327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "a5d84185dda64117b3d793e359da9dd8"
format_stamp: "Formatted at 2026-08-06 10:47:17 on dist-test-slave-0b58"
server_key: "92a1edfe1e9cdf19e53ae729823bacc3"
server_key_iv: "d35cb62093c568d03a9f27b872e187f7"
server_key_version: "encryptionkey@0"
I20260806 10:47:17.303803 18327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:17.316692 18327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:17.321326 18327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:17.321621 18327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:17.324155 18327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:17.324337 18327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:17.324471 18327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:17.324565 18327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:47:17 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013237, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:17.351438 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.325004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:41173 (local address 127.16.21.62:39517)
0806 10:47:17.326433 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:17.326442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:17.327247 (+ 805us) server_negotiation.cc:407] Connection header received
0806 10:47:17.330214 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:17.330221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:17.330322 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:17.330426 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:17.332828 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.333892 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.337212 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.337749 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.337917 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.342955 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:17.342964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:17.342972 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:17.343018 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:17.347017 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.348286 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.348293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.348297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.348455 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.349126 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.349134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.349140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.349599 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:17.349762 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:17.350949 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0806 10:47:17.351118 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":1115,"threads_started":1}
I20260806 10:47:17.353240 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.325770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.16.21.1:41173)
0806 10:47:17.327065 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:17.327112 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:17.329870 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:17.330693 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:17.330708 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:17.331543 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:17.332595 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.332630 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.334059 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.334064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.337060 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.337075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.338116 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.338128 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.338488 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.339225 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:17.339262 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:17.342722 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:17.347197 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.347221 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.347241 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.348043 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.348621 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.348629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.348633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.348962 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.349862 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:17.349870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:17.350421 (+ 551us) client_negotiation.cc:769] Sending connection context
0806 10:47:17.350789 (+ 368us) client_negotiation.cc:240] Negotiation successful
0806 10:47:17.351458 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":792,"threads_started":1}
I20260806 10:47:17.375058 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.359663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:45851 (local address 127.16.21.62:39517)
0806 10:47:17.359891 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:17.359900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:17.360517 (+ 617us) server_negotiation.cc:407] Connection header received
0806 10:47:17.360684 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:17.360689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:17.360764 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:17.360864 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:17.362439 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.363216 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.365955 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.366951 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.367107 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.369164 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:17.369190 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:17.369198 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:17.369241 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:17.372320 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.372989 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.372994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.372998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.373131 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.373573 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.373581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.373586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.373936 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:17.374114 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:17.374511 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 10:47:17.374737 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:47:17.375146 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.359593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.16.21.1:45851)
0806 10:47:17.360400 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:17.360421 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:17.360567 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:17.360888 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:17.360892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:17.361746 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:17.362260 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.362277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.363385 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.363392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.365717 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.365736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.367138 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.367143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.367280 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.367914 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:17.367930 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:17.368911 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:17.372454 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.372460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.372463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.372865 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.373240 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.373244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.373248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.373375 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.374063 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:17.374068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:17.374252 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 10:47:17.374388 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 10:47:17.374690 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":419,"threads_started":1}
I20260806 10:47:17.385077 18327 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:39949
I20260806 10:47:17.385126 18465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:39949 every 8 connection(s)
I20260806 10:47:17.386210 18327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:17.386722 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 18327
I20260806 10:47:17.386952 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:17.387841 16468 external_mini_cluster.cc:1428] Setting key b88bc7d434b6f533cf10cd03a81186e9
I20260806 10:47:17.388466 18466 heartbeater.cc:347] Connected to a master server at 127.16.21.62:39517
I20260806 10:47:17.388797 18466 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:17.389690 18466 heartbeater.cc:511] Master 127.16.21.62:39517 requested a full tablet report, sending...
I20260806 10:47:17.391716 18255 ts_manager.cc:194] Registered new tserver with Master: a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949)
I20260806 10:47:17.393177 18255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:41173
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:17.435433 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:39517
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:17.750968 18471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:17.751230 18471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:17.751435 18471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:17.758767 18471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:17.758884 18471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:17.759038 18471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 10:47:17.759117 18471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:47:17.759248 18471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:17.768589 18471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:17.769716 18471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:17.770944 18471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:17.784457 18479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:17.784433 18477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:17.785089 18476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:17.785619 18471 server_base.cc:1034] running on GCE node
I20260806 10:47:17.786176 18471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:17.787097 18471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:17.788342 18471 hybrid_clock.cc:648] HybridClock initialized: now 1786013237788294 us; error 42 us; skew 500 ppm
I20260806 10:47:17.788640 18471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:17 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013237, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:17.791630 18471 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:17.793056 18471 webserver.cc:460] Webserver started at http://127.16.21.2:36509/ using document root <none> and password file <none>
I20260806 10:47:17.793632 18471 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:17.793730 18471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:17.793974 18471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:17.796727 18471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "51807008b6e0442ea692ecb4c84b8363"
format_stamp: "Formatted at 2026-08-06 10:47:17 on dist-test-slave-0b58"
server_key: "287d4b5a1433fa0d5a60575c53f9ecc9"
server_key_iv: "b509c9a0b99e613225dad045a4f173f0"
server_key_version: "encryptionkey@0"
I20260806 10:47:17.797396 18471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "51807008b6e0442ea692ecb4c84b8363"
format_stamp: "Formatted at 2026-08-06 10:47:17 on dist-test-slave-0b58"
server_key: "287d4b5a1433fa0d5a60575c53f9ecc9"
server_key_iv: "b509c9a0b99e613225dad045a4f173f0"
server_key_version: "encryptionkey@0"
I20260806 10:47:17.802132 18471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:47:17.805266 18486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:17.806465 18471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:17.806583 18471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "51807008b6e0442ea692ecb4c84b8363"
format_stamp: "Formatted at 2026-08-06 10:47:17 on dist-test-slave-0b58"
server_key: "287d4b5a1433fa0d5a60575c53f9ecc9"
server_key_iv: "b509c9a0b99e613225dad045a4f173f0"
server_key_version: "encryptionkey@0"
I20260806 10:47:17.806736 18471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:17.829404 18471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:17.833428 18471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:17.833678 18471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:17.836184 18471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:17.836334 18471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:17.836483 18471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:17.836549 18471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:47:17 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013237, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:17.862393 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.836608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:49241 (local address 127.16.21.62:39517)
0806 10:47:17.836872 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:17.836882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:17.838369 (+ 1487us) server_negotiation.cc:407] Connection header received
0806 10:47:17.841544 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:17.841551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:17.841690 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:17.841803 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:17.844639 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.845620 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.848936 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.849544 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.849692 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.854800 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:17.854810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:17.854819 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:17.854871 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:17.858786 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.859751 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.859756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.859760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.859895 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.860463 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.860468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.860471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.860870 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:17.861021 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:17.861815 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 10:47:17.862044 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 10:47:17.863499 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.837244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.16.21.2:49241)
0806 10:47:17.838190 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:17.838237 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:17.841180 (+ 2943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:17.842389 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:17.842403 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:17.843359 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:17.844397 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.844434 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.845837 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.845845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.848748 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.848772 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.849865 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.849876 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.850227 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.851156 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:17.851195 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:17.854505 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:17.858913 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.858921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.858935 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.859614 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.860030 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.860036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.860040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.860328 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.861081 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:17.861087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:17.861410 (+ 323us) client_negotiation.cc:769] Sending connection context
0806 10:47:17.861678 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 10:47:17.862165 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":391,"threads_started":1}
I20260806 10:47:17.887642 18471 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:46263
I20260806 10:47:17.887727 18609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:46263 every 8 connection(s)
I20260806 10:47:17.889099 18471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:17.891296 18610 heartbeater.cc:347] Connected to a master server at 127.16.21.62:39517
I20260806 10:47:17.891601 18610 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:17.892432 18610 heartbeater.cc:511] Master 127.16.21.62:39517 requested a full tablet report, sending...
I20260806 10:47:17.892853 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.872391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:52939 (local address 127.16.21.62:39517)
0806 10:47:17.872598 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:17.872604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:17.877072 (+ 4468us) server_negotiation.cc:407] Connection header received
0806 10:47:17.877290 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:17.877296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:17.877376 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:17.877481 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:17.878881 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.879741 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.882516 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.883286 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.883449 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.886133 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:17.886138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:17.886141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:17.886180 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:17.889337 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.890428 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.890433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.890436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.890607 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:17.891092 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:17.891097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:17.891099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:17.891591 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:17.891745 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:17.892382 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 10:47:17.892596 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:47:17.893043 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:17.872781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.16.21.2:52939)
0806 10:47:17.876930 (+ 4149us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:17.876958 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:17.877128 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:17.877557 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:17.877564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:17.878343 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:17.878738 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.878756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.879938 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.879947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.882331 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:17.882354 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:17.883534 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:17.883544 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:17.883781 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:17.884639 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:17.884672 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:17.885954 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:17.889456 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.889461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.889465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.890256 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.890773 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:17.890781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:17.890788 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:17.890956 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:17.891827 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:17.891833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:17.892040 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 10:47:17.892202 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 10:47:17.892563 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3923,"thread_start_us":3708,"threads_started":1}
I20260806 10:47:17.893612 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 18471
I20260806 10:47:17.893796 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:17.894155 18254 ts_manager.cc:194] Registered new tserver with Master: 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263)
I20260806 10:47:17.894369 16468 external_mini_cluster.cc:1428] Setting key 025761703e19d027704a7d7679d3c6e3
I20260806 10:47:17.895388 18254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:49241
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:17.939898 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:39517
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:18.267077 18615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:18.267366 18615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:18.267542 18615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:18.274546 18615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:18.274688 18615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:18.274824 18615 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 10:47:18.274911 18615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:47:18.275041 18615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:18.284569 18615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:18.285780 18615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:18.287001 18615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:18.299721 18620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:18.299752 18623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:18.299912 18621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:18.301443 18615 server_base.cc:1034] running on GCE node
I20260806 10:47:18.301944 18615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:18.302734 18615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:18.303936 18615 hybrid_clock.cc:648] HybridClock initialized: now 1786013238303884 us; error 49 us; skew 500 ppm
I20260806 10:47:18.304239 18615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:18 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:18.312739 18615 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:18.314100 18615 webserver.cc:460] Webserver started at http://127.16.21.3:38165/ using document root <none> and password file <none>
I20260806 10:47:18.314747 18615 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:18.314903 18615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:18.315254 18615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:18.318619 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6"
format_stamp: "Formatted at 2026-08-06 10:47:18 on dist-test-slave-0b58"
server_key: "00445ad51178926abe806b650390d767"
server_key_iv: "af0a8c8a87e0e6288620da240c54f2ff"
server_key_version: "encryptionkey@0"
I20260806 10:47:18.319429 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6"
format_stamp: "Formatted at 2026-08-06 10:47:18 on dist-test-slave-0b58"
server_key: "00445ad51178926abe806b650390d767"
server_key_iv: "af0a8c8a87e0e6288620da240c54f2ff"
server_key_version: "encryptionkey@0"
I20260806 10:47:18.324232 18615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 10:47:18.327585 18630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:18.328907 18615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:47:18.329072 18615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6"
format_stamp: "Formatted at 2026-08-06 10:47:18 on dist-test-slave-0b58"
server_key: "00445ad51178926abe806b650390d767"
server_key_iv: "af0a8c8a87e0e6288620da240c54f2ff"
server_key_version: "encryptionkey@0"
I20260806 10:47:18.329255 18615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:18.355316 18615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:18.359193 18615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:18.359458 18615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:18.362116 18615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:18.362257 18615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:18.362406 18615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:18.362528 18615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 06 10:47:18 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:18.392531 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.362737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:50137 (local address 127.16.21.62:39517)
0806 10:47:18.363615 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.363623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.364174 (+ 551us) server_negotiation.cc:407] Connection header received
0806 10:47:18.367005 (+ 2831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.367014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.367125 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.367247 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:18.369471 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.370591 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.373249 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.373690 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.373831 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.379678 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:18.379687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:18.379695 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:18.379745 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:18.383277 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.384409 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.384418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.384424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.384621 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.388745 (+ 4124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.388753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.388757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.390048 (+ 1291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:18.390219 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.391626 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.391965 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716}
I20260806 10:47:18.393080 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.363132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.16.21.3:50137)
0806 10:47:18.363987 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.364031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.366722 (+ 2691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.367529 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.367541 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.368355 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:18.369216 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.369245 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.370790 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.370794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.373099 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.373118 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.373952 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.373960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.374357 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.375145 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:18.375186 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:18.379351 (+ 4165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:18.383454 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:18.383475 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:18.383501 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:18.384236 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:18.384776 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:18.384782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:18.384785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:18.388588 (+ 3803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:18.390180 (+ 1592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:18.390189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:18.390697 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.391060 (+ 363us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.391712 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":432,"threads_started":1}
I20260806 10:47:18.396335 18466 heartbeater.cc:503] Master 127.16.21.62:39517 was elected leader, sending a full tablet report...
I20260806 10:47:18.420239 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.403912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:53669 (local address 127.16.21.62:39517)
0806 10:47:18.404135 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.404144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.405015 (+ 871us) server_negotiation.cc:407] Connection header received
0806 10:47:18.405223 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.405229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.405331 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.405411 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:18.406934 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.408021 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.409780 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.410357 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.410506 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.413373 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:18.413381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:18.413389 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:18.413439 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:18.417362 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.418118 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.418124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.418127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.418282 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.418656 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.418661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.418665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.418941 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:18.419115 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.419637 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.419859 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:47:18.420337 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.403908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.16.21.3:53669)
0806 10:47:18.404869 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.404905 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.405081 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.405488 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.405496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.406280 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:18.406753 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.406767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.408164 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.408170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.409636 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.409649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.410760 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.410766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.411058 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.411872 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:18.411901 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:18.413131 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:18.417483 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:18.417489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:18.417493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:18.418006 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:18.418399 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:18.418404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:18.418407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:18.418540 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:18.419168 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:18.419172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:18.419397 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.419529 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.419843 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":585,"threads_started":1}
I20260806 10:47:18.426076 18615 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:46581
I20260806 10:47:18.426126 18752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:46581 every 8 connection(s)
I20260806 10:47:18.427295 18615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:18.427743 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 18615
I20260806 10:47:18.427932 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:18.428606 16468 external_mini_cluster.cc:1428] Setting key 2a6e70ff3b52b84094aa414f29bafd4d
I20260806 10:47:18.429611 18753 heartbeater.cc:347] Connected to a master server at 127.16.21.62:39517
I20260806 10:47:18.429873 18753 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:18.430497 18753 heartbeater.cc:511] Master 127.16.21.62:39517 requested a full tablet report, sending...
I20260806 10:47:18.431823 18255 ts_manager.cc:194] Registered new tserver with Master: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581)
I20260806 10:47:18.432735 18255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:50137
I20260806 10:47:18.443696 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:47:18 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:47:18 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:18.497143 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.483050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.16.21.62:39517)
0806 10:47:18.483335 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.483342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.483570 (+ 228us) server_negotiation.cc:407] Connection header received
0806 10:47:18.483729 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.483733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.483799 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.483886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:18.485310 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.486071 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.487323 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.487776 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.487876 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.491626 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:18.491631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:18.491634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:18.491665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:18.494437 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.495153 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.495157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.495160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.495294 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.495641 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.495645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.495647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.495953 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:18.496140 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.496530 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.496758 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 10:47:18.501022 18255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:18.508680 18255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:18.533393 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.520787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.0.0.1:56698)
0806 10:47:18.521561 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.521610 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.521824 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.523470 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.523482 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.523524 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.524013 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.524029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.526822 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.526831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.529927 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.529949 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.532349 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.532359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.532635 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.532702 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.532839 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.533039 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":307,"threads_started":1}
I20260806 10:47:18.534170 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.521603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.16.21.3:46581)
0806 10:47:18.522309 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.522318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.522351 (+ 33us) server_negotiation.cc:407] Connection header received
0806 10:47:18.522461 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.522468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.522738 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.522983 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.524375 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.526606 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.530174 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.531790 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.531932 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.532179 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.533657 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.533821 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":384,"threads_started":1}
I20260806 10:47:18.537711 18681 tablet_service.cc:1467] Processing CreateTablet for tablet a5d5430cc1c84264baa46e2d69406a7e (DEFAULT_TABLE table=test-table [id=58abaafb00864ed6afbb9809b5e58f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:18.539192 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d5430cc1c84264baa46e2d69406a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:18.541592 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.522996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:39949 (local address 127.0.0.1:40130)
0806 10:47:18.532803 (+ 9807us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.532824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.532927 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.534544 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.534548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.534574 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.534958 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.534972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.536844 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.536849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.538941 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.538957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.540816 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.540824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.540977 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.541009 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.541107 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.541240 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9672,"thread_start_us":9563,"threads_started":1}
I20260806 10:47:18.542379 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.523767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.16.21.1:39949)
0806 10:47:18.532233 (+ 8466us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.532324 (+ 91us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.533038 (+ 714us) server_negotiation.cc:407] Connection header received
0806 10:47:18.533137 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.533146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.533393 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.533648 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.535146 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.536701 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.539114 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.540592 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.540733 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.540927 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.541166 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.541344 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8296,"thread_start_us":8134,"threads_started":1}
I20260806 10:47:18.542395 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.521034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46263 (local address 127.0.0.1:44114)
0806 10:47:18.530709 (+ 9675us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.530732 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.530851 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.535762 (+ 4911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.535767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.535805 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.536138 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.536149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.537887 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.537892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.539757 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.539783 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.541564 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.541573 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.541771 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.541815 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.541880 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.542006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9556,"mutex_wait_us":2,"thread_start_us":9874,"threads_started":1}
I20260806 10:47:18.543327 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.521291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.16.21.2:46263)
0806 10:47:18.535181 (+ 13890us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.535188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.535215 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:47:18.535310 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.535317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.535559 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.536349 (+ 790us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.536534 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.537708 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.539952 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.541328 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.542637 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.542823 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.542893 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.543060 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13723,"thread_start_us":13586,"threads_started":1}
I20260806 10:47:18.545975 18538 tablet_service.cc:1467] Processing CreateTablet for tablet a5d5430cc1c84264baa46e2d69406a7e (DEFAULT_TABLE table=test-table [id=58abaafb00864ed6afbb9809b5e58f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:18.546190 18394 tablet_service.cc:1467] Processing CreateTablet for tablet a5d5430cc1c84264baa46e2d69406a7e (DEFAULT_TABLE table=test-table [id=58abaafb00864ed6afbb9809b5e58f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:18.547343 18538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d5430cc1c84264baa46e2d69406a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:18.547482 18394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d5430cc1c84264baa46e2d69406a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:18.553962 18772 tablet_bootstrap.cc:492] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Bootstrap starting.
I20260806 10:47:18.557662 18772 tablet_bootstrap.cc:654] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:18.558941 18772 log.cc:824] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:18.559777 18773 tablet_bootstrap.cc:492] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Bootstrap starting.
I20260806 10:47:18.560784 18774 tablet_bootstrap.cc:492] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Bootstrap starting.
I20260806 10:47:18.562794 18773 tablet_bootstrap.cc:654] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:18.564013 18773 log.cc:824] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:18.564505 18774 tablet_bootstrap.cc:654] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:18.565645 18772 tablet_bootstrap.cc:492] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: No bootstrap required, opened a new log
I20260806 10:47:18.565810 18774 log.cc:824] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:18.565935 18772 ts_tablet_manager.cc:1397] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 10:47:18.567140 18773 tablet_bootstrap.cc:492] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: No bootstrap required, opened a new log
I20260806 10:47:18.567399 18773 ts_tablet_manager.cc:1397] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:47:18.568749 18774 tablet_bootstrap.cc:492] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: No bootstrap required, opened a new log
I20260806 10:47:18.569003 18774 ts_tablet_manager.cc:1397] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260806 10:47:18.574607 18773 raft_consensus.cc:348] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.574968 18773 raft_consensus.cc:374] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:18.575067 18773 raft_consensus.cc:729] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Initialized, Role: FOLLOWER
I20260806 10:47:18.575664 18773 consensus_queue.cc:260] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.576761 18610 heartbeater.cc:503] Master 127.16.21.62:39517 was elected leader, sending a full tablet report...
I20260806 10:47:18.577289 18773 ts_tablet_manager.cc:1428] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260806 10:47:18.576534 18772 raft_consensus.cc:348] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.577852 18772 raft_consensus.cc:374] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:18.577975 18772 raft_consensus.cc:729] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Initialized, Role: FOLLOWER
I20260806 10:47:18.578534 18772 consensus_queue.cc:260] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.579751 18774 raft_consensus.cc:348] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.580143 18774 raft_consensus.cc:374] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:18.580298 18774 raft_consensus.cc:729] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d84185dda64117b3d793e359da9dd8, State: Initialized, Role: FOLLOWER
I20260806 10:47:18.580933 18774 consensus_queue.cc:260] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.581640 18753 heartbeater.cc:503] Master 127.16.21.62:39517 was elected leader, sending a full tablet report...
I20260806 10:47:18.582250 18772 ts_tablet_manager.cc:1428] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260806 10:47:18.583808 18774 ts_tablet_manager.cc:1428] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260806 10:47:18.640686 18468 tablet.cc:2369] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:18.643131 18611 tablet.cc:2369] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:18.667689 18778 raft_consensus.cc:482] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:18.668043 18778 raft_consensus.cc:504] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.669739 18778 leader_election.cc:288] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581), a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949)
I20260806 10:47:18.676862 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.670221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45957 (local address 127.16.21.3:46581)
0806 10:47:18.670664 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.670673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.671357 (+ 684us) server_negotiation.cc:407] Connection header received
0806 10:47:18.671531 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.671537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.671618 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.671715 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.672396 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.673152 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.675017 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.675787 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.675886 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.676024 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.676479 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.676600 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260806 10:47:18.676863 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.670358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:54901 (local address 127.16.21.1:39949)
0806 10:47:18.670624 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.670634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.671534 (+ 900us) server_negotiation.cc:407] Connection header received
0806 10:47:18.671717 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.671722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.671824 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.671928 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.672556 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.673265 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.675048 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.675815 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.675999 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.676111 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.676466 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.676601 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:47:18.677004 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.670008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.16.21.2:45957)
0806 10:47:18.671241 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.671258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.671382 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.671743 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.671749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.671775 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.672201 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.672212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.673296 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.673303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.674856 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.674868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.676041 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.676047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.676300 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.676324 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.676431 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.676552 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":34,"thread_start_us":826,"threads_started":1}
I20260806 10:47:18.677241 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.670164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:39949 (local address 127.16.21.2:54901)
0806 10:47:18.671390 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.671409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.671564 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.671962 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.671966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.671986 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.672398 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.672410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.673408 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.673412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.674873 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.674885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.676084 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.676089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.676210 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.676307 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.676391 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.676540 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":1,"thread_start_us":749,"threads_started":1}
I20260806 10:47:18.678076 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d5430cc1c84264baa46e2d69406a7e" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" is_pre_election: true
I20260806 10:47:18.678391 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d5430cc1c84264baa46e2d69406a7e" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8" is_pre_election: true
I20260806 10:47:18.678568 18701 raft_consensus.cc:2454] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 0.
I20260806 10:47:18.678892 18419 raft_consensus.cc:2454] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 0.
W20260806 10:47:18.679081 18754 tablet.cc:2369] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:18.679399 18487 leader_election.cc:302] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, 7b9cca9c9e1e4dc0bc7292dbf5dfaea6; no voters:
I20260806 10:47:18.679831 18778 raft_consensus.cc:2781] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:18.680034 18778 raft_consensus.cc:482] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:18.680141 18778 raft_consensus.cc:3037] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:18.682716 18778 raft_consensus.cc:504] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.683332 18778 leader_election.cc:288] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 election: Requested vote from peers 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581), a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949)
I20260806 10:47:18.683736 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d5430cc1c84264baa46e2d69406a7e" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6"
I20260806 10:47:18.683830 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d5430cc1c84264baa46e2d69406a7e" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8"
I20260806 10:47:18.683941 18701 raft_consensus.cc:3037] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:18.684068 18419 raft_consensus.cc:3037] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:18.686271 18701 raft_consensus.cc:2454] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 1.
I20260806 10:47:18.686408 18419 raft_consensus.cc:2454] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 1.
I20260806 10:47:18.686668 18487 leader_election.cc:302] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, 7b9cca9c9e1e4dc0bc7292dbf5dfaea6; no voters:
I20260806 10:47:18.686949 18778 raft_consensus.cc:2781] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:18.687386 18778 raft_consensus.cc:686] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 1 LEADER]: Becoming Leader. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Running, Role: LEADER
I20260806 10:47:18.687847 18778 consensus_queue.cc:237] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } }
I20260806 10:47:18.692749 18255 catalog_manager.cc:5495] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "51807008b6e0442ea692ecb4c84b8363" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } health_report { overall_health: UNKNOWN } } }
Aug 06 10:47:18 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013236, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:18.763572 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.750566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:39517 (local address 127.0.0.1:52856)
0806 10:47:18.751185 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.751200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.751330 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.751622 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.751625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.752339 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:18.752612 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.752621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.753598 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.753602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.754574 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.754585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.755262 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.755267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.755443 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.756167 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:18.756191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:18.758389 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:18.761429 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:18.761438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:18.761443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:18.761847 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:18.762166 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:18.762169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:18.762171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:18.762308 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:18.762772 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:18.762778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:18.762907 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.763047 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.763238 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":337,"threads_started":1}
I20260806 10:47:18.763604 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.750555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.16.21.62:39517)
0806 10:47:18.750793 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.750801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.751294 (+ 493us) server_negotiation.cc:407] Connection header received
0806 10:47:18.751435 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.751438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.751497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.751578 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:18.752755 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.753487 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.754696 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.755101 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.755192 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.758555 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:18.758559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:18.758562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:18.758590 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:18.761316 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.761946 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.761949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.761952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.762074 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:18.762397 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:18.762400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:18.762402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:18.762654 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:18.762761 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.763153 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.763345 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":77}
I20260806 10:47:18.768529 18255 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:52856:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 10:47:18.769447 18255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:18.776309 18394 tablet_service.cc:1467] Processing CreateTablet for tablet 99363722efc4475aa2d523a1da569f68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acdb6ce59beb4e4d9c6616e16b850096]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:47:18.776489 18538 tablet_service.cc:1467] Processing CreateTablet for tablet 99363722efc4475aa2d523a1da569f68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acdb6ce59beb4e4d9c6616e16b850096]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:47:18.777184 18394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99363722efc4475aa2d523a1da569f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:18.777354 18538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99363722efc4475aa2d523a1da569f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:18.783115 18774 tablet_bootstrap.cc:492] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: Bootstrap starting.
I20260806 10:47:18.783864 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 99363722efc4475aa2d523a1da569f68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acdb6ce59beb4e4d9c6616e16b850096]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:47:18.784726 18774 tablet_bootstrap.cc:654] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:18.784754 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99363722efc4475aa2d523a1da569f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:18.785687 18773 tablet_bootstrap.cc:492] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: Bootstrap starting.
I20260806 10:47:18.787205 18773 tablet_bootstrap.cc:654] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:18.787204 18774 tablet_bootstrap.cc:492] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: No bootstrap required, opened a new log
I20260806 10:47:18.787352 18774 ts_tablet_manager.cc:1397] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 10:47:18.788206 18774 raft_consensus.cc:348] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.788558 18774 raft_consensus.cc:374] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:18.788674 18774 raft_consensus.cc:729] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d84185dda64117b3d793e359da9dd8, State: Initialized, Role: FOLLOWER
I20260806 10:47:18.789022 18774 consensus_queue.cc:260] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.789603 18781 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:18.790068 18781 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:18.791837 18772 tablet_bootstrap.cc:492] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Bootstrap starting.
I20260806 10:47:18.793401 18772 tablet_bootstrap.cc:654] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:18.793668 18774 ts_tablet_manager.cc:1428] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 10:47:18.797180 18772 tablet_bootstrap.cc:492] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: No bootstrap required, opened a new log
I20260806 10:47:18.797338 18772 ts_tablet_manager.cc:1397] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260806 10:47:18.798055 18772 raft_consensus.cc:348] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.798298 18772 raft_consensus.cc:374] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:18.798379 18772 raft_consensus.cc:729] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Initialized, Role: FOLLOWER
I20260806 10:47:18.798645 18772 consensus_queue.cc:260] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.799190 18780 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:18.799690 18780 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:18.799207 18772 ts_tablet_manager.cc:1428] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:47:18.802055 18773 tablet_bootstrap.cc:492] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: No bootstrap required, opened a new log
I20260806 10:47:18.802208 18773 ts_tablet_manager.cc:1397] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.014s
I20260806 10:47:18.803022 18773 raft_consensus.cc:348] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.803282 18773 raft_consensus.cc:374] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:18.803417 18773 raft_consensus.cc:729] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Initialized, Role: FOLLOWER
I20260806 10:47:18.803659 18773 consensus_queue.cc:260] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.804101 18778 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:18.804402 18778 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:18.804916 18773 ts_tablet_manager.cc:1428] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 10:47:18.970423 18780 raft_consensus.cc:482] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:18.970645 18780 raft_consensus.cc:504] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.972165 18780 leader_election.cc:288] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949), 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263)
I20260806 10:47:18.980551 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.972870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:47791 (local address 127.16.21.2:46263)
0806 10:47:18.973113 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.973121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.973487 (+ 366us) server_negotiation.cc:407] Connection header received
0806 10:47:18.973610 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.973618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.973709 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.973792 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.974371 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.975439 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.978234 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.979476 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.979621 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.979773 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.980146 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.980308 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:47:18.980576 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.972498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:44595 (local address 127.16.21.1:39949)
0806 10:47:18.972716 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:18.972726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:18.973233 (+ 507us) server_negotiation.cc:407] Connection header received
0806 10:47:18.973429 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:18.973433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:18.973552 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:18.973672 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:18.974198 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.975404 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.978234 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.979501 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.979621 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.979782 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:18.980150 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 10:47:18.980318 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:47:18.980621 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.972467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46263 (local address 127.16.21.3:47791)
0806 10:47:18.973370 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.973392 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.973489 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.973829 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.973833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.973857 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.974159 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.974168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.975573 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.975579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.978067 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.978084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.979717 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.979722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.979966 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.979990 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.980095 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.980222 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":103,"thread_start_us":628,"threads_started":1}
I20260806 10:47:18.981341 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:18.972398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:39949 (local address 127.16.21.3:44595)
0806 10:47:18.973099 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:18.973120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:18.973284 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:18.973712 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:18.973717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:18.973739 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:18.974017 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.974034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.975573 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.975579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.978067 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:18.978084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:18.979672 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:18.979680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:18.979891 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:18.979970 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 10:47:18.980813 (+ 843us) client_negotiation.cc:240] Negotiation successful
0806 10:47:18.980935 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":211,"thread_start_us":447,"threads_started":1}
I20260806 10:47:18.981815 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99363722efc4475aa2d523a1da569f68" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51807008b6e0442ea692ecb4c84b8363" is_pre_election: true
I20260806 10:47:18.981957 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99363722efc4475aa2d523a1da569f68" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8" is_pre_election: true
I20260806 10:47:18.982223 18419 raft_consensus.cc:2454] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 0.
I20260806 10:47:18.982221 18562 raft_consensus.cc:2454] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 0.
I20260806 10:47:18.982735 18633 leader_election.cc:302] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, a5d84185dda64117b3d793e359da9dd8; no voters:
I20260806 10:47:18.983162 18780 raft_consensus.cc:2781] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:18.983310 18780 raft_consensus.cc:482] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:18.983425 18780 raft_consensus.cc:3037] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:18.985605 18780 raft_consensus.cc:504] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.986197 18780 leader_election.cc:288] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 election: Requested vote from peers a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949), 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263)
I20260806 10:47:18.986657 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99363722efc4475aa2d523a1da569f68" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8"
I20260806 10:47:18.986673 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99363722efc4475aa2d523a1da569f68" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51807008b6e0442ea692ecb4c84b8363"
I20260806 10:47:18.986874 18562 raft_consensus.cc:3037] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:18.986893 18419 raft_consensus.cc:3037] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:18.989895 18419 raft_consensus.cc:2454] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 1.
I20260806 10:47:18.989964 18562 raft_consensus.cc:2454] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 1.
I20260806 10:47:18.990372 18634 leader_election.cc:302] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, a5d84185dda64117b3d793e359da9dd8; no voters:
I20260806 10:47:18.990715 18780 raft_consensus.cc:2781] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:18.991174 18780 raft_consensus.cc:686] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 LEADER]: Becoming Leader. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Running, Role: LEADER
I20260806 10:47:18.991703 18780 consensus_queue.cc:237] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:18.994843 18797 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: TxnStatusTablet state changed. Reason: New leader 7b9cca9c9e1e4dc0bc7292dbf5dfaea6. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:18.995303 18797 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:47:18.995965 18799 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 10:47:18.996301 18799 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 10:47:18.996328 18255 catalog_manager.cc:5495] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } health_report { overall_health: HEALTHY } } }
I20260806 10:47:19.025218 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.019129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.0.0.1:56712)
0806 10:47:19.019421 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:19.019441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:19.019586 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:19.020068 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:19.020073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:19.020087 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:19.020369 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:19.020383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.021463 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.021471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:19.022522 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:19.022534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.023412 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.023419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:19.023623 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.023682 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.024654 (+ 972us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.024681 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:47:19.024772 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:47:19.024919 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 10:47:19.025223 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.019204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.16.21.3:46581)
0806 10:47:19.019426 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:19.019435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:19.019495 (+ 60us) server_negotiation.cc:407] Connection header received
0806 10:47:19.019797 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:19.019801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:19.019894 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:19.020031 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:19.020510 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.021327 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.022673 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.023246 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.023389 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.023879 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.024535 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.024692 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:19.024785 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 10:47:19.024868 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260806 10:47:19.033354 18562 raft_consensus.cc:1261] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Refusing update from remote peer 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:19.033356 18419 raft_consensus.cc:1261] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Refusing update from remote peer 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:19.034332 18780 consensus_queue.cc:1035] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:19.034686 18797 consensus_queue.cc:1035] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:19.042925 18781 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: TxnStatusTablet state changed. Reason: New leader 7b9cca9c9e1e4dc0bc7292dbf5dfaea6. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:19.043200 18781 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:19.045102 18802 mvcc.cc:205] Tried to move back new op lower bound from 7315510227070332928 to 7315510226913779712. Current Snapshot: MvccSnapshot[applied={T|T < 7315510227070332928}]
I20260806 10:47:19.046876 18797 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:19.047142 18797 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:47:19.053553 18781 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:19.053828 18781 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P a5d84185dda64117b3d793e359da9dd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:19.057277 18778 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: TxnStatusTablet state changed. Reason: New leader 7b9cca9c9e1e4dc0bc7292dbf5dfaea6. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:19.057679 18778 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:19.060833 18780 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:19.061196 18780 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:47:19.068027 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.057354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.16.21.2:46263)
0806 10:47:19.057548 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:19.057556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:19.060248 (+ 2692us) server_negotiation.cc:407] Connection header received
0806 10:47:19.060609 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:19.060615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:19.060721 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:19.060848 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:19.061527 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.062640 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.064811 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.065550 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.065646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.066585 (+ 939us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.067211 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.067347 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:19.067616 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 10:47:19.067702 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:47:19.069530 18778 tablet_replica.cc:439] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } } }
I20260806 10:47:19.069831 18778 tablet_replica.cc:442] T 99363722efc4475aa2d523a1da569f68 P 51807008b6e0442ea692ecb4c84b8363: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:47:19.084648 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.078515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.16.21.2:49663)
0806 10:47:19.079154 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:19.079174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:19.079281 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:19.079645 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:19.079648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:19.079658 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:19.079996 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:19.080008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.081532 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.081535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:19.082607 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:19.082618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.083575 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.083579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:19.083690 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.083737 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.084154 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.084161 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:47:19.084219 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 10:47:19.084377 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":334,"threads_started":1}
I20260806 10:47:19.084672 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.078510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:49663 (local address 127.16.21.3:46581)
0806 10:47:19.078699 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:19.078706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:19.079254 (+ 548us) server_negotiation.cc:407] Connection header received
0806 10:47:19.079414 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:19.079420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:19.079527 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:19.079621 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:19.080159 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.081403 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.082741 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.083390 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.083508 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.083924 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.084056 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.084147 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:19.084297 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 10:47:19.084391 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:47:19.093912 18419 raft_consensus.cc:1261] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Refusing update from remote peer 51807008b6e0442ea692ecb4c84b8363: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:19.094463 18701 raft_consensus.cc:1261] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Refusing update from remote peer 51807008b6e0442ea692ecb4c84b8363: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:19.094553 18784 consensus_queue.cc:1035] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:19.095144 18778 consensus_queue.cc:1035] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:19.130016 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.123134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:59279 (local address 127.16.21.2:46263)
0806 10:47:19.123324 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:19.123332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:19.123930 (+ 598us) server_negotiation.cc:407] Connection header received
0806 10:47:19.124143 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:19.124149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:19.124236 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:19.124356 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:19.124961 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.126103 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.127815 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.128510 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.128632 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.129262 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.129403 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.129462 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:19.129667 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 10:47:19.129747 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:47:19.130837 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:19.122859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46263 (local address 127.16.21.3:59279)
0806 10:47:19.123828 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:19.123844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:19.123986 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:19.124385 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:19.124389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:19.124398 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:19.124793 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:19.124806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.126262 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.126266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:19.127649 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:19.127661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:19.128798 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:19.128818 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:19.128999 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:19.129068 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:19.129542 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:19.129552 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:47:19.130295 (+ 743us) client_negotiation.cc:240] Negotiation successful
0806 10:47:19.130433 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":706,"threads_started":1}
I20260806 10:47:19.169915 18254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:19.170215 18254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:19.174213 18254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Sending DeleteTablet for 3 replicas of tablet a5d5430cc1c84264baa46e2d69406a7e
I20260806 10:47:19.175396 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d5430cc1c84264baa46e2d69406a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:56698
I20260806 10:47:19.175520 18394 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d5430cc1c84264baa46e2d69406a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260806 10:47:19.176468 18825 tablet_replica.cc:330] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: stopping tablet replica
I20260806 10:47:19.176827 18538 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d5430cc1c84264baa46e2d69406a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260806 10:47:19.176946 18825 raft_consensus.cc:2229] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:19.177457 18825 raft_consensus.cc:2258] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:19.179030 18254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:19.179738 18254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:19.185240 18826 tablet_replica.cc:330] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: stopping tablet replica
I20260806 10:47:19.185755 18826 raft_consensus.cc:2229] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:19.187402 18826 raft_consensus.cc:2258] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:19.187908 18537 tablet_service.cc:1467] Processing CreateTablet for tablet cf2abc3e9cda4badb5d35ef15215f8e0 (DEFAULT_TABLE table=test-table [id=43e479e55b8a4a3485b427bfe49c6597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:19.188489 18537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2abc3e9cda4badb5d35ef15215f8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:19.191095 18825 ts_tablet_manager.cc:1905] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:19.192718 18827 tablet_replica.cc:330] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: stopping tablet replica
I20260806 10:47:19.192919 18681 tablet_service.cc:1467] Processing CreateTablet for tablet cf2abc3e9cda4badb5d35ef15215f8e0 (DEFAULT_TABLE table=test-table [id=43e479e55b8a4a3485b427bfe49c6597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:19.193171 18827 raft_consensus.cc:2229] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:19.193392 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2abc3e9cda4badb5d35ef15215f8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:19.193655 18826 ts_tablet_manager.cc:1905] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:19.193831 18827 raft_consensus.cc:2258] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:19.195618 18394 tablet_service.cc:1467] Processing CreateTablet for tablet cf2abc3e9cda4badb5d35ef15215f8e0 (DEFAULT_TABLE table=test-table [id=43e479e55b8a4a3485b427bfe49c6597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:19.196074 18394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2abc3e9cda4badb5d35ef15215f8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:19.196385 18827 ts_tablet_manager.cc:1905] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:19.204339 18827 ts_tablet_manager.cc:1918] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:19.204555 18827 log.cc:1196] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/a5d5430cc1c84264baa46e2d69406a7e
I20260806 10:47:19.205188 18827 ts_tablet_manager.cc:1939] T a5d5430cc1c84264baa46e2d69406a7e P 51807008b6e0442ea692ecb4c84b8363: Deleting consensus metadata
I20260806 10:47:19.207145 18241 catalog_manager.cc:4841] TS 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263): tablet a5d5430cc1c84264baa46e2d69406a7e (table test-table [id=58abaafb00864ed6afbb9809b5e58f78]) successfully deleted
I20260806 10:47:19.209967 18773 tablet_bootstrap.cc:492] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Bootstrap starting.
I20260806 10:47:19.211882 18773 tablet_bootstrap.cc:654] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:19.214185 18825 ts_tablet_manager.cc:1918] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:19.214393 18825 log.cc:1196] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/a5d5430cc1c84264baa46e2d69406a7e
I20260806 10:47:19.214886 18772 tablet_bootstrap.cc:492] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Bootstrap starting.
I20260806 10:47:19.216666 18772 tablet_bootstrap.cc:654] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:19.218770 18825 ts_tablet_manager.cc:1939] T a5d5430cc1c84264baa46e2d69406a7e P a5d84185dda64117b3d793e359da9dd8: Deleting consensus metadata
I20260806 10:47:19.220372 18774 tablet_bootstrap.cc:492] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Bootstrap starting.
I20260806 10:47:19.220942 18240 catalog_manager.cc:4841] TS a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949): tablet a5d5430cc1c84264baa46e2d69406a7e (table test-table [id=58abaafb00864ed6afbb9809b5e58f78]) successfully deleted
I20260806 10:47:19.221989 18774 tablet_bootstrap.cc:654] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:19.224589 18773 tablet_bootstrap.cc:492] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: No bootstrap required, opened a new log
I20260806 10:47:19.224812 18773 ts_tablet_manager.cc:1397] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260806 10:47:19.225661 18773 raft_consensus.cc:348] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.226022 18773 raft_consensus.cc:374] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:19.226186 18773 raft_consensus.cc:729] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Initialized, Role: FOLLOWER
I20260806 10:47:19.226522 18773 consensus_queue.cc:260] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.227977 18773 ts_tablet_manager.cc:1428] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:19.228771 18826 ts_tablet_manager.cc:1918] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:19.229034 18826 log.cc:1196] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/a5d5430cc1c84264baa46e2d69406a7e
I20260806 10:47:19.232075 18826 ts_tablet_manager.cc:1939] T a5d5430cc1c84264baa46e2d69406a7e P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting consensus metadata
I20260806 10:47:19.233916 18238 catalog_manager.cc:4841] TS 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581): tablet a5d5430cc1c84264baa46e2d69406a7e (table test-table [id=58abaafb00864ed6afbb9809b5e58f78]) successfully deleted
I20260806 10:47:19.237545 18774 tablet_bootstrap.cc:492] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: No bootstrap required, opened a new log
I20260806 10:47:19.237715 18774 ts_tablet_manager.cc:1397] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20260806 10:47:19.238412 18774 raft_consensus.cc:348] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.238638 18774 raft_consensus.cc:374] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:19.238706 18774 raft_consensus.cc:729] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d84185dda64117b3d793e359da9dd8, State: Initialized, Role: FOLLOWER
I20260806 10:47:19.238996 18774 consensus_queue.cc:260] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.240094 18772 tablet_bootstrap.cc:492] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: No bootstrap required, opened a new log
I20260806 10:47:19.240370 18772 ts_tablet_manager.cc:1397] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.008s
I20260806 10:47:19.241253 18772 raft_consensus.cc:348] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.241568 18772 raft_consensus.cc:374] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:19.241703 18772 raft_consensus.cc:729] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Initialized, Role: FOLLOWER
I20260806 10:47:19.241707 18774 ts_tablet_manager.cc:1428] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260806 10:47:19.242018 18772 consensus_queue.cc:260] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.243268 18772 ts_tablet_manager.cc:1428] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:19.525620 18778 raft_consensus.cc:482] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:19.525864 18778 raft_consensus.cc:504] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.526746 18778 leader_election.cc:288] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949), 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581)
I20260806 10:47:19.527359 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2abc3e9cda4badb5d35ef15215f8e0" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8" is_pre_election: true
I20260806 10:47:19.527614 18419 raft_consensus.cc:2454] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 0.
I20260806 10:47:19.527594 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2abc3e9cda4badb5d35ef15215f8e0" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" is_pre_election: true
I20260806 10:47:19.527881 18701 raft_consensus.cc:2454] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 0.
I20260806 10:47:19.528337 18489 leader_election.cc:302] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, a5d84185dda64117b3d793e359da9dd8; no voters:
I20260806 10:47:19.528708 18778 raft_consensus.cc:2781] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:19.528856 18778 raft_consensus.cc:482] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:19.528959 18778 raft_consensus.cc:3037] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:19.532232 18778 raft_consensus.cc:504] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.532893 18778 leader_election.cc:288] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 election: Requested vote from peers a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949), 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581)
I20260806 10:47:19.533468 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2abc3e9cda4badb5d35ef15215f8e0" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8"
I20260806 10:47:19.533661 18419 raft_consensus.cc:3037] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:19.533630 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2abc3e9cda4badb5d35ef15215f8e0" candidate_uuid: "51807008b6e0442ea692ecb4c84b8363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6"
I20260806 10:47:19.533857 18701 raft_consensus.cc:3037] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:19.536657 18419 raft_consensus.cc:2454] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 1.
I20260806 10:47:19.537101 18489 leader_election.cc:302] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, a5d84185dda64117b3d793e359da9dd8; no voters:
I20260806 10:47:19.537420 18778 raft_consensus.cc:2781] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:19.537426 18701 raft_consensus.cc:2454] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51807008b6e0442ea692ecb4c84b8363 in term 1.
I20260806 10:47:19.537931 18778 raft_consensus.cc:686] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 1 LEADER]: Becoming Leader. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Running, Role: LEADER
I20260806 10:47:19.538237 18778 consensus_queue.cc:237] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:19.541013 18254 catalog_manager.cc:5495] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "51807008b6e0442ea692ecb4c84b8363" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } health_report { overall_health: UNKNOWN } } }
I20260806 10:47:19.573720 18701 raft_consensus.cc:1261] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Refusing update from remote peer 51807008b6e0442ea692ecb4c84b8363: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:19.573988 18418 raft_consensus.cc:1261] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Refusing update from remote peer 51807008b6e0442ea692ecb4c84b8363: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:19.574617 18819 consensus_queue.cc:1035] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:19.575006 18778 consensus_queue.cc:1035] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:19.586637 18804 mvcc.cc:205] Tried to move back new op lower bound from 7315510229287460864 to 7315510229150773248. Current Snapshot: MvccSnapshot[applied={T|T < 7315510229287460864}]
W20260806 10:47:19.608345 18804 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 10:47:21.190964 18255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:21.191215 18255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:21.195232 18255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Sending DeleteTablet for 3 replicas of tablet cf2abc3e9cda4badb5d35ef15215f8e0
I20260806 10:47:21.196022 18394 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2abc3e9cda4badb5d35ef15215f8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:21 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260806 10:47:21.196130 18538 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2abc3e9cda4badb5d35ef15215f8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:21 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260806 10:47:21.196449 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2abc3e9cda4badb5d35ef15215f8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:21 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:56698
I20260806 10:47:21.206874 18856 tablet_replica.cc:330] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: stopping tablet replica
I20260806 10:47:21.207242 18856 raft_consensus.cc:2229] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:21.207437 18858 tablet_replica.cc:330] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: stopping tablet replica
I20260806 10:47:21.207675 18858 raft_consensus.cc:2229] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:21.207854 18856 raft_consensus.cc:2258] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:21.207933 18858 raft_consensus.cc:2258] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:21.208758 18857 tablet_replica.cc:330] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: stopping tablet replica
I20260806 10:47:21.208953 18857 raft_consensus.cc:2229] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:21.209154 18858 ts_tablet_manager.cc:1905] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:21.209416 18857 raft_consensus.cc:2258] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:21.210420 18857 ts_tablet_manager.cc:1905] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:21.215946 18858 ts_tablet_manager.cc:1918] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:47:21.216107 18858 log.cc:1196] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/cf2abc3e9cda4badb5d35ef15215f8e0
I20260806 10:47:21.216812 18856 ts_tablet_manager.cc:1905] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:21.217046 18857 ts_tablet_manager.cc:1918] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:47:21.217173 18857 log.cc:1196] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/cf2abc3e9cda4badb5d35ef15215f8e0
I20260806 10:47:21.217690 18857 ts_tablet_manager.cc:1939] T cf2abc3e9cda4badb5d35ef15215f8e0 P 51807008b6e0442ea692ecb4c84b8363: Deleting consensus metadata
I20260806 10:47:21.218765 18241 catalog_manager.cc:4841] TS 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263): tablet cf2abc3e9cda4badb5d35ef15215f8e0 (table test-table [id=43e479e55b8a4a3485b427bfe49c6597]) successfully deleted
I20260806 10:47:21.220543 18858 ts_tablet_manager.cc:1939] T cf2abc3e9cda4badb5d35ef15215f8e0 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting consensus metadata
I20260806 10:47:21.221653 18238 catalog_manager.cc:4841] TS 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581): tablet cf2abc3e9cda4badb5d35ef15215f8e0 (table test-table [id=43e479e55b8a4a3485b427bfe49c6597]) successfully deleted
I20260806 10:47:21.223410 18856 ts_tablet_manager.cc:1918] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:47:21.223538 18856 log.cc:1196] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/cf2abc3e9cda4badb5d35ef15215f8e0
I20260806 10:47:21.223969 18856 ts_tablet_manager.cc:1939] T cf2abc3e9cda4badb5d35ef15215f8e0 P a5d84185dda64117b3d793e359da9dd8: Deleting consensus metadata
I20260806 10:47:21.225016 18240 catalog_manager.cc:4841] TS a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949): tablet cf2abc3e9cda4badb5d35ef15215f8e0 (table test-table [id=43e479e55b8a4a3485b427bfe49c6597]) successfully deleted
Aug 06 10:47:21 dist-test-slave-0b58 krb5kdc[18208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:47:21 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:21.266664 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.251740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.16.21.62:39517)
0806 10:47:21.252436 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.252441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.252470 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:47:21.252609 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.252621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.252693 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.252813 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:21.254231 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.255208 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.256643 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.257051 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.257188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.261250 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:21.261256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:21.261259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:21.261291 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:21.263819 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:21.264575 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:21.264587 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:21.264589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:21.264783 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:21.265193 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:21.265197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:21.265201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:21.265476 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:21.265755 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.266067 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.266311 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":328,"threads_started":1}
I20260806 10:47:21.270085 18255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:21.270829 18255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:21.277539 18394 tablet_service.cc:1467] Processing CreateTablet for tablet 4caddac775774179b925332279f8a928 (DEFAULT_TABLE table=test-table [id=a6ebf6a349494e7b9ad427aa9ab03af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:21.277909 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 4caddac775774179b925332279f8a928 (DEFAULT_TABLE table=test-table [id=a6ebf6a349494e7b9ad427aa9ab03af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:21.278015 18394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caddac775774179b925332279f8a928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:21.277941 18538 tablet_service.cc:1467] Processing CreateTablet for tablet 4caddac775774179b925332279f8a928 (DEFAULT_TABLE table=test-table [id=a6ebf6a349494e7b9ad427aa9ab03af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:21.278328 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caddac775774179b925332279f8a928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:21.278364 18538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caddac775774179b925332279f8a928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:21.283756 18867 tablet_bootstrap.cc:492] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Bootstrap starting.
I20260806 10:47:21.285094 18869 tablet_bootstrap.cc:492] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Bootstrap starting.
I20260806 10:47:21.285465 18867 tablet_bootstrap.cc:654] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:21.287168 18868 tablet_bootstrap.cc:492] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Bootstrap starting.
I20260806 10:47:21.287213 18869 tablet_bootstrap.cc:654] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:21.287928 18867 tablet_bootstrap.cc:492] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: No bootstrap required, opened a new log
I20260806 10:47:21.288080 18867 ts_tablet_manager.cc:1397] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 10:47:21.289111 18868 tablet_bootstrap.cc:654] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:21.289027 18867 raft_consensus.cc:348] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.289304 18867 raft_consensus.cc:374] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:21.289405 18867 raft_consensus.cc:729] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Initialized, Role: FOLLOWER
I20260806 10:47:21.289729 18867 consensus_queue.cc:260] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.289937 18869 tablet_bootstrap.cc:492] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: No bootstrap required, opened a new log
I20260806 10:47:21.290084 18869 ts_tablet_manager.cc:1397] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260806 10:47:21.290270 18867 ts_tablet_manager.cc:1428] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 10:47:21.291013 18869 raft_consensus.cc:348] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.291337 18869 raft_consensus.cc:374] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:21.291489 18869 raft_consensus.cc:729] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Initialized, Role: FOLLOWER
I20260806 10:47:21.291823 18869 consensus_queue.cc:260] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.292093 18868 tablet_bootstrap.cc:492] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: No bootstrap required, opened a new log
I20260806 10:47:21.292301 18868 ts_tablet_manager.cc:1397] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260806 10:47:21.293061 18869 ts_tablet_manager.cc:1428] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 10:47:21.293134 18868 raft_consensus.cc:348] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.293390 18868 raft_consensus.cc:374] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:21.293497 18868 raft_consensus.cc:729] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d84185dda64117b3d793e359da9dd8, State: Initialized, Role: FOLLOWER
I20260806 10:47:21.293785 18868 consensus_queue.cc:260] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.294706 18868 ts_tablet_manager.cc:1428] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 10:47:21.516376 18873 raft_consensus.cc:482] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:21.516690 18873 raft_consensus.cc:504] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.518697 18873 leader_election.cc:288] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263), 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581)
I20260806 10:47:21.527539 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.519656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:39087 (local address 127.16.21.3:46581)
0806 10:47:21.520678 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.520687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.521289 (+ 602us) server_negotiation.cc:407] Connection header received
0806 10:47:21.521504 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.521512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.521583 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.521673 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:21.522198 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.523307 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.525732 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.526527 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.526676 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.526792 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.527055 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.527185 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":630,"threads_started":1}
I20260806 10:47:21.527532 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.519083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.16.21.1:39087)
0806 10:47:21.521197 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:21.521210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:21.521340 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:21.521699 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:21.521704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:21.521731 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:21.522036 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.522046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.523563 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.523568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.525590 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.525602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.526721 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.526727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.526895 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.526919 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:47:21.527031 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 10:47:21.527159 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1990,"thread_start_us":1794,"threads_started":1}
I20260806 10:47:21.528378 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caddac775774179b925332279f8a928" candidate_uuid: "a5d84185dda64117b3d793e359da9dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" is_pre_election: true
I20260806 10:47:21.528658 18701 raft_consensus.cc:2454] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d84185dda64117b3d793e359da9dd8 in term 0.
I20260806 10:47:21.529163 18343 leader_election.cc:302] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, a5d84185dda64117b3d793e359da9dd8; no voters:
I20260806 10:47:21.529598 18873 raft_consensus.cc:2781] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:21.529702 18873 raft_consensus.cc:482] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:21.529773 18873 raft_consensus.cc:3037] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:21.530526 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.518853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46263 (local address 127.16.21.1:58721)
0806 10:47:21.519704 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:21.519733 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:21.519952 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:21.522933 (+ 2981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:21.522940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:21.522975 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:21.523513 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.523530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.525080 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.525088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.527626 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.527645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.529413 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.529422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.529674 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.529741 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 10:47:21.529855 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 10:47:21.530042 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":1516,"threads_started":1}
I20260806 10:47:21.531522 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.519312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:58721 (local address 127.16.21.2:46263)
0806 10:47:21.522584 (+ 3272us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.522589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.522612 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:47:21.522681 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.522686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.522756 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.522908 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:21.523744 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.524925 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.527915 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.529151 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.529294 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.529459 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.530933 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.531114 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3134,"thread_start_us":2982,"threads_started":1}
I20260806 10:47:21.531934 18873 raft_consensus.cc:504] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.532124 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caddac775774179b925332279f8a928" candidate_uuid: "a5d84185dda64117b3d793e359da9dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51807008b6e0442ea692ecb4c84b8363" is_pre_election: true
I20260806 10:47:21.532384 18562 raft_consensus.cc:2454] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d84185dda64117b3d793e359da9dd8 in term 0.
I20260806 10:47:21.532691 18873 leader_election.cc:288] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [CANDIDATE]: Term 1 election: Requested vote from peers 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263), 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581)
I20260806 10:47:21.533286 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caddac775774179b925332279f8a928" candidate_uuid: "a5d84185dda64117b3d793e359da9dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6"
I20260806 10:47:21.533504 18701 raft_consensus.cc:3037] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:21.533454 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caddac775774179b925332279f8a928" candidate_uuid: "a5d84185dda64117b3d793e359da9dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51807008b6e0442ea692ecb4c84b8363"
I20260806 10:47:21.533676 18562 raft_consensus.cc:3037] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:21.535580 18701 raft_consensus.cc:2454] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d84185dda64117b3d793e359da9dd8 in term 1.
I20260806 10:47:21.535991 18343 leader_election.cc:302] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, a5d84185dda64117b3d793e359da9dd8; no voters:
I20260806 10:47:21.536298 18873 raft_consensus.cc:2781] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:21.536814 18873 raft_consensus.cc:686] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 1 LEADER]: Becoming Leader. State: Replica: a5d84185dda64117b3d793e359da9dd8, State: Running, Role: LEADER
I20260806 10:47:21.537297 18562 raft_consensus.cc:2454] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d84185dda64117b3d793e359da9dd8 in term 1.
I20260806 10:47:21.537235 18873 consensus_queue.cc:237] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.540922 18253 catalog_manager.cc:5495] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d84185dda64117b3d793e359da9dd8 (127.16.21.1). New cstate: current_term: 1 leader_uuid: "a5d84185dda64117b3d793e359da9dd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } health_report { overall_health: UNKNOWN } } }
I20260806 10:47:21.600117 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.592287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.16.21.1:39949)
0806 10:47:21.593051 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.593059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.593088 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:47:21.593199 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.593207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.593329 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.593535 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:21.593853 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.595367 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.596856 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.597701 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.597880 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.598313 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.599260 (+ 947us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.599430 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.599488 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.599589 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":444,"threads_started":1}
I20260806 10:47:21.621364 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.611029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.16.21.1:50333)
0806 10:47:21.615225 (+ 4196us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:21.615240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:21.615357 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:21.615820 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:21.615827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:21.615841 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:21.616181 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.616190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.617853 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.617856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.619020 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.619037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.620228 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.620233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.620429 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.620482 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.620909 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.620917 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:47:21.620970 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 10:47:21.621064 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4062,"thread_start_us":3863,"threads_started":1}
I20260806 10:47:21.621497 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.611019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:50333 (local address 127.16.21.3:46581)
0806 10:47:21.611262 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.611270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.615329 (+ 4059us) server_negotiation.cc:407] Connection header received
0806 10:47:21.615514 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.615523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.615631 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.615768 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:21.616367 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.617714 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.619165 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.620031 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.620177 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.620650 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.620805 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.620904 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.621040 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.621138 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:47:21.631258 18701 raft_consensus.cc:1261] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Refusing update from remote peer a5d84185dda64117b3d793e359da9dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:21.631304 18562 raft_consensus.cc:1261] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Refusing update from remote peer a5d84185dda64117b3d793e359da9dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:21.631946 18878 consensus_queue.cc:1035] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:21.632367 18873 consensus_queue.cc:1035] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:21.637521 18839 mvcc.cc:205] Tried to move back new op lower bound from 7315510237715132416 to 7315510237340119040. Current Snapshot: MvccSnapshot[applied={T|T < 7315510237715132416}]
I20260806 10:47:21.667713 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.660629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:50243 (local address 127.16.21.1:39949)
0806 10:47:21.660824 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.660834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.661496 (+ 662us) server_negotiation.cc:407] Connection header received
0806 10:47:21.661675 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.661678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.661737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.661820 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:21.662367 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.663464 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.665619 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.666335 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.666453 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.666957 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.667089 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.667185 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.667423 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.667499 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 10:47:21.667904 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.660596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:39949 (local address 127.16.21.3:50243)
0806 10:47:21.661380 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:21.661402 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:21.661571 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:21.661860 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:21.661868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:21.661885 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:21.662187 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.662197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.663652 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.663658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.665432 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.665453 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.666600 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.666605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.666734 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.666770 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.667298 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.667310 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 10:47:21.667386 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:47:21.667525 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":382,"threads_started":1}
I20260806 10:47:21.703173 18253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:21.703447 18253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:21.706575 18253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Sending DeleteTablet for 3 replicas of tablet 4caddac775774179b925332279f8a928
I20260806 10:47:21.707343 18394 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caddac775774179b925332279f8a928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:21 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260806 10:47:21.707506 18538 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caddac775774179b925332279f8a928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:21 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260806 10:47:21.707520 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caddac775774179b925332279f8a928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:21 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:56698
I20260806 10:47:21.707657 18856 tablet_replica.cc:330] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: stopping tablet replica
I20260806 10:47:21.707790 18858 tablet_replica.cc:330] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: stopping tablet replica
I20260806 10:47:21.707837 18857 tablet_replica.cc:330] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: stopping tablet replica
I20260806 10:47:21.707858 18856 raft_consensus.cc:2229] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:21.707989 18858 raft_consensus.cc:2229] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:21.708022 18857 raft_consensus.cc:2229] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:21.708191 18858 raft_consensus.cc:2258] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:21.708211 18857 raft_consensus.cc:2258] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:21.708401 18856 raft_consensus.cc:2258] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:21.709625 18858 ts_tablet_manager.cc:1905] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:21.709636 18856 ts_tablet_manager.cc:1905] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:21.709767 18857 ts_tablet_manager.cc:1905] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:21.716486 18253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 10:47:21.716706 18856 ts_tablet_manager.cc:1918] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:21.716857 18856 log.cc:1196] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/4caddac775774179b925332279f8a928
W20260806 10:47:21.717165 18253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:21.717444 18856 ts_tablet_manager.cc:1939] T 4caddac775774179b925332279f8a928 P a5d84185dda64117b3d793e359da9dd8: Deleting consensus metadata
I20260806 10:47:21.717480 18858 ts_tablet_manager.cc:1918] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:21.717623 18858 log.cc:1196] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/4caddac775774179b925332279f8a928
I20260806 10:47:21.718088 18858 ts_tablet_manager.cc:1939] T 4caddac775774179b925332279f8a928 P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting consensus metadata
I20260806 10:47:21.718539 18240 catalog_manager.cc:4841] TS a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949): tablet 4caddac775774179b925332279f8a928 (table test-table [id=a6ebf6a349494e7b9ad427aa9ab03af5]) successfully deleted
I20260806 10:47:21.718823 18857 ts_tablet_manager.cc:1918] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:47:21.718962 18857 log.cc:1196] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/4caddac775774179b925332279f8a928
I20260806 10:47:21.719262 18238 catalog_manager.cc:4841] TS 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581): tablet 4caddac775774179b925332279f8a928 (table test-table [id=a6ebf6a349494e7b9ad427aa9ab03af5]) successfully deleted
I20260806 10:47:21.719409 18857 ts_tablet_manager.cc:1939] T 4caddac775774179b925332279f8a928 P 51807008b6e0442ea692ecb4c84b8363: Deleting consensus metadata
I20260806 10:47:21.720433 18241 catalog_manager.cc:4841] TS 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263): tablet 4caddac775774179b925332279f8a928 (table test-table [id=a6ebf6a349494e7b9ad427aa9ab03af5]) successfully deleted
I20260806 10:47:21.725152 18394 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a (DEFAULT_TABLE table=test-table [id=c6a49ceb7d6945289447f327368d2ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:21.725560 18394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:21.725592 18538 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a (DEFAULT_TABLE table=test-table [id=c6a49ceb7d6945289447f327368d2ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:21.726014 18538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:21.730255 18868 tablet_bootstrap.cc:492] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Bootstrap starting.
I20260806 10:47:21.730983 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a (DEFAULT_TABLE table=test-table [id=c6a49ceb7d6945289447f327368d2ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:21.731410 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:21.731974 18868 tablet_bootstrap.cc:654] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:21.732369 18869 tablet_bootstrap.cc:492] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Bootstrap starting.
I20260806 10:47:21.733968 18869 tablet_bootstrap.cc:654] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:21.739557 18867 tablet_bootstrap.cc:492] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Bootstrap starting.
I20260806 10:47:21.741685 18867 tablet_bootstrap.cc:654] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:21.750516 18867 tablet_bootstrap.cc:492] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: No bootstrap required, opened a new log
I20260806 10:47:21.750825 18867 ts_tablet_manager.cc:1397] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260806 10:47:21.751876 18867 raft_consensus.cc:348] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.752540 18867 raft_consensus.cc:374] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:21.752707 18867 raft_consensus.cc:729] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Initialized, Role: FOLLOWER
I20260806 10:47:21.753135 18867 consensus_queue.cc:260] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.754828 18867 ts_tablet_manager.cc:1428] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260806 10:47:21.756503 18869 tablet_bootstrap.cc:492] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: No bootstrap required, opened a new log
I20260806 10:47:21.756768 18869 ts_tablet_manager.cc:1397] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.017s
I20260806 10:47:21.757660 18869 raft_consensus.cc:348] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.757995 18868 tablet_bootstrap.cc:492] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: No bootstrap required, opened a new log
I20260806 10:47:21.758000 18869 raft_consensus.cc:374] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:21.758087 18869 raft_consensus.cc:729] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51807008b6e0442ea692ecb4c84b8363, State: Initialized, Role: FOLLOWER
I20260806 10:47:21.758288 18868 ts_tablet_manager.cc:1397] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.014s
I20260806 10:47:21.758473 18869 consensus_queue.cc:260] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.759145 18869 ts_tablet_manager.cc:1428] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:47:21.759197 18868 raft_consensus.cc:348] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.759476 18868 raft_consensus.cc:374] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:21.759613 18868 raft_consensus.cc:729] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d84185dda64117b3d793e359da9dd8, State: Initialized, Role: FOLLOWER
I20260806 10:47:21.759908 18868 consensus_queue.cc:260] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.760732 18868 ts_tablet_manager.cc:1428] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 10:47:21.804402 18837 raft_consensus.cc:482] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:21.804630 18837 raft_consensus.cc:504] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.805380 18837 leader_election.cc:288] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949), 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263)
I20260806 10:47:21.805898 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca7fd3eda1c4239bf1123ed15cbbd7a" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8" is_pre_election: true
I20260806 10:47:21.806007 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca7fd3eda1c4239bf1123ed15cbbd7a" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51807008b6e0442ea692ecb4c84b8363" is_pre_election: true
I20260806 10:47:21.806236 18419 raft_consensus.cc:2454] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 0.
I20260806 10:47:21.806262 18562 raft_consensus.cc:2454] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 0.
I20260806 10:47:21.806672 18634 leader_election.cc:302] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, 7b9cca9c9e1e4dc0bc7292dbf5dfaea6; no voters:
I20260806 10:47:21.806982 18837 raft_consensus.cc:2781] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:21.807128 18837 raft_consensus.cc:482] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:21.807226 18837 raft_consensus.cc:3037] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:21.810415 18837 raft_consensus.cc:504] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.811070 18837 leader_election.cc:288] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 election: Requested vote from peers a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949), 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263)
I20260806 10:47:21.811599 18562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca7fd3eda1c4239bf1123ed15cbbd7a" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51807008b6e0442ea692ecb4c84b8363"
I20260806 10:47:21.811816 18562 raft_consensus.cc:3037] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:21.814899 18562 raft_consensus.cc:2454] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 1.
I20260806 10:47:21.815146 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca7fd3eda1c4239bf1123ed15cbbd7a" candidate_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d84185dda64117b3d793e359da9dd8"
I20260806 10:47:21.815418 18419 raft_consensus.cc:3037] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:21.815582 18634 leader_election.cc:302] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51807008b6e0442ea692ecb4c84b8363, 7b9cca9c9e1e4dc0bc7292dbf5dfaea6; no voters:
I20260806 10:47:21.815904 18837 raft_consensus.cc:2781] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:21.816118 18837 raft_consensus.cc:686] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 LEADER]: Becoming Leader. State: Replica: 7b9cca9c9e1e4dc0bc7292dbf5dfaea6, State: Running, Role: LEADER
I20260806 10:47:21.816531 18837 consensus_queue.cc:237] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } }
I20260806 10:47:21.819281 18419 raft_consensus.cc:2454] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 in term 1.
I20260806 10:47:21.821964 18253 catalog_manager.cc:5495] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b9cca9c9e1e4dc0bc7292dbf5dfaea6" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 46581 } health_report { overall_health: HEALTHY } } }
I20260806 10:47:21.857412 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.852029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.16.21.3:46581)
0806 10:47:21.852387 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.852392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.852661 (+ 269us) server_negotiation.cc:407] Connection header received
0806 10:47:21.852867 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.852872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.852949 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.853043 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:21.853447 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.854203 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.855582 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.856091 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.856223 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.856646 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.856749 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.856849 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.857002 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.857083 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260806 10:47:21.869884 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.864890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:48411 (local address 127.16.21.3:46581)
0806 10:47:21.865035 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:21.865039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:21.865053 (+ 14us) server_negotiation.cc:407] Connection header received
0806 10:47:21.865246 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:21.865249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:21.865319 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:21.865387 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:21.865770 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.866520 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.868370 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.868754 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.868840 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.869289 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.869387 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.869446 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:21.869620 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 10:47:21.869673 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 10:47:21.870127 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:21.864734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:46581 (local address 127.16.21.3:48411)
0806 10:47:21.864938 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:21.864975 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:21.865075 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:21.865436 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:21.865441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:21.865454 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:47:21.865631 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.865641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.866650 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.866656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.868186 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:21.868200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:21.868898 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:21.868905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:21.869093 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:21.869137 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:47:21.869479 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:47:21.869492 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 10:47:21.869596 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 10:47:21.869750 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 10:47:21.878011 18562 raft_consensus.cc:1261] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Refusing update from remote peer 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:21.878057 18419 raft_consensus.cc:1261] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Refusing update from remote peer 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:21.878473 18837 consensus_queue.cc:1035] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51807008b6e0442ea692ecb4c84b8363" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:21.878809 18842 consensus_queue.cc:1035] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d84185dda64117b3d793e359da9dd8" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 10:47:21.907148 18835 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 10:47:23.500067 18255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:23.500332 18255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:23.504400 18255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b2e684469bb4d1a832783b994d4b35b: Sending DeleteTablet for 3 replicas of tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a
I20260806 10:47:23.505147 18394 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:23 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260806 10:47:23.505180 18538 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:23 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260806 10:47:23.505214 18680 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:23 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:56698
I20260806 10:47:23.505928 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18327
I20260806 10:47:23.517705 18921 tablet_replica.cc:330] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: stopping tablet replica
I20260806 10:47:23.517952 18923 tablet_replica.cc:330] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: stopping tablet replica
I20260806 10:47:23.518321 18923 raft_consensus.cc:2229] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:23.518479 18921 raft_consensus.cc:2229] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:23.518965 18923 raft_consensus.cc:2258] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:23.519093 18922 tablet_replica.cc:330] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: stopping tablet replica
I20260806 10:47:23.519179 18921 raft_consensus.cc:2258] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:23.519385 18922 raft_consensus.cc:2229] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:23.519701 18922 raft_consensus.cc:2258] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:23.520519 18923 ts_tablet_manager.cc:1905] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:23.520715 18921 ts_tablet_manager.cc:1905] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:23.520846 18922 ts_tablet_manager.cc:1905] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:23.527514 18923 ts_tablet_manager.cc:1918] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:47:23.527580 18922 ts_tablet_manager.cc:1918] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:47:23.527612 18921 ts_tablet_manager.cc:1918] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:47:23.527660 18923 log.cc:1196] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/0ca7fd3eda1c4239bf1123ed15cbbd7a
I20260806 10:47:23.527743 18921 log.cc:1196] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/0ca7fd3eda1c4239bf1123ed15cbbd7a
I20260806 10:47:23.527822 18922 log.cc:1196] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/0ca7fd3eda1c4239bf1123ed15cbbd7a
I20260806 10:47:23.528213 18923 ts_tablet_manager.cc:1939] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 7b9cca9c9e1e4dc0bc7292dbf5dfaea6: Deleting consensus metadata
I20260806 10:47:23.528293 18921 ts_tablet_manager.cc:1939] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P a5d84185dda64117b3d793e359da9dd8: Deleting consensus metadata
I20260806 10:47:23.528312 18922 ts_tablet_manager.cc:1939] T 0ca7fd3eda1c4239bf1123ed15cbbd7a P 51807008b6e0442ea692ecb4c84b8363: Deleting consensus metadata
I20260806 10:47:23.529258 18241 catalog_manager.cc:4841] TS 51807008b6e0442ea692ecb4c84b8363 (127.16.21.2:46263): tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a (table test-table [id=c6a49ceb7d6945289447f327368d2ee0]) successfully deleted
I20260806 10:47:23.529379 18240 catalog_manager.cc:4841] TS a5d84185dda64117b3d793e359da9dd8 (127.16.21.1:39949): tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a (table test-table [id=c6a49ceb7d6945289447f327368d2ee0]) successfully deleted
I20260806 10:47:23.529609 18238 catalog_manager.cc:4841] TS 7b9cca9c9e1e4dc0bc7292dbf5dfaea6 (127.16.21.3:46581): tablet 0ca7fd3eda1c4239bf1123ed15cbbd7a (table test-table [id=c6a49ceb7d6945289447f327368d2ee0]) successfully deleted
Aug 06 10:47:23 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:23.539347 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:23.510050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.16.21.1:39949)
0806 10:47:23.518888 (+ 8838us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:23.518894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:23.518920 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:47:23.519005 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:23.519012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:23.519094 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:23.519267 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:23.521205 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:23.522424 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:23.524589 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:23.525251 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:23.525390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:23.534165 (+ 8775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:23.534181 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:23.534196 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:23.534233 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:23.536913 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:23.537513 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:23.537520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:23.537522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:23.537656 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:23.537980 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:23.537983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:23.537985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:23.538326 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:23.538549 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:23.538788 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 10:47:23.538948 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8692,"thread_start_us":8543,"threads_started":1}
W20260806 10:47:23.540022 18460 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45640
W20260806 10:47:23.540725 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:23.540943 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 18327
I20260806 10:47:23.568163 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18471
Aug 06 10:47:23 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:23.582969 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:23.568993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.16.21.2:46263)
0806 10:47:23.569542 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:23.569547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:23.569562 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:47:23.569616 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:23.569620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:23.569676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:23.569773 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:23.570944 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:23.571650 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:23.572914 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:23.573312 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:23.573406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:23.577493 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:23.577509 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:23.577524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:23.577561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:23.580738 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:23.581317 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:23.581324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:23.581326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:23.581447 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:23.581753 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:23.581773 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:23.581775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:23.582100 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:23.582336 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:23.582543 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 10:47:23.582709 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
W20260806 10:47:23.583642 18605 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55598
W20260806 10:47:23.584208 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18471: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:23.584394 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 18471
I20260806 10:47:23.611552 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18615
Aug 06 10:47:23 dist-test-slave-0b58 krb5kdc[18208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:23.626318 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:23.612221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.16.21.3:46581)
0806 10:47:23.612822 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:23.612826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:23.612842 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:47:23.612896 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:23.612899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:23.612954 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:23.613041 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:23.614252 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:23.614976 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:23.616385 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:23.616797 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:23.616894 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:23.621180 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:23.621195 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:23.621211 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:23.621242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:23.624002 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:23.624667 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:23.624674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:23.624676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:23.624804 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:23.625118 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:23.625121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:23.625123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:23.625446 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:23.625665 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:23.625866 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 10:47:23.626018 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":342,"threads_started":1}
W20260806 10:47:23.626983 18747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35192
W20260806 10:47:23.627640 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:23.627817 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 18615
I20260806 10:47:23.658164 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18222
I20260806 10:47:23.659346 18296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:23.779016 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 18222
2026-08-06T10:47:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8772 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:23 dist-test-slave-0b58 krb5kdc[18931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:23 dist-test-slave-0b58 krb5kdc[18931](info): set up 2 sockets
Aug 06 10:47:23 dist-test-slave-0b58 krb5kdc[18931](info): commencing operation
krb5kdc: starting...
W20260806 10:47:24.879927 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:25 dist-test-slave-0b58 krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:25.089949 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:42117
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:42117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:25.430581 18944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:25.430922 18944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:25.431098 18944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:25.438463 18944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:25.438563 18944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:25.438658 18944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:25.438771 18944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:25.438850 18944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:25.448906 18944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:42117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:42117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:25.450208 18944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:25.451543 18944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:25.463824 18949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:25.463892 18950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:25.464375 18944 server_base.cc:1034] running on GCE node
W20260806 10:47:25.464563 18952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:25.465461 18944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:25.466861 18944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:25.468065 18944 hybrid_clock.cc:648] HybridClock initialized: now 1786013245468033 us; error 27 us; skew 500 ppm
I20260806 10:47:25.468423 18944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:25 dist-test-slave-0b58 krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:25.471398 18944 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:25.472827 18944 webserver.cc:460] Webserver started at http://127.16.21.62:40705/ using document root <none> and password file <none>
I20260806 10:47:25.473449 18944 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:25.473548 18944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:25.473816 18944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:25.476711 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "0580b03ed60b4a9ebd33046d22e116bb"
format_stamp: "Formatted at 2026-08-06 10:47:25 on dist-test-slave-0b58"
server_key: "478831df3a1209aa1eaf9588554bfd76"
server_key_iv: "d45f810eaabb4fd7366bb07db386f19c"
server_key_version: "encryptionkey@0"
I20260806 10:47:25.477413 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "0580b03ed60b4a9ebd33046d22e116bb"
format_stamp: "Formatted at 2026-08-06 10:47:25 on dist-test-slave-0b58"
server_key: "478831df3a1209aa1eaf9588554bfd76"
server_key_iv: "d45f810eaabb4fd7366bb07db386f19c"
server_key_version: "encryptionkey@0"
I20260806 10:47:25.482474 18944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:47:25.485924 18959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:25.487223 18944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:47:25.487404 18944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "0580b03ed60b4a9ebd33046d22e116bb"
format_stamp: "Formatted at 2026-08-06 10:47:25 on dist-test-slave-0b58"
server_key: "478831df3a1209aa1eaf9588554bfd76"
server_key_iv: "d45f810eaabb4fd7366bb07db386f19c"
server_key_version: "encryptionkey@0"
I20260806 10:47:25.487582 18944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:25.504393 18944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:25.508338 18944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:25.508626 18944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:25.528002 18944 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:42117
I20260806 10:47:25.528021 19011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:42117 every 8 connection(s)
I20260806 10:47:25.529436 18944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:25.533144 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:25.538146 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 18944
I20260806 10:47:25.538388 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:25.539064 16468 external_mini_cluster.cc:1428] Setting key 6da21bf5103823803485bfa27f61d75c
I20260806 10:47:25.543958 19012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb: Bootstrap starting.
I20260806 10:47:25.547714 19012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:25.548985 19012 log.cc:824] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:25.552428 19012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb: No bootstrap required, opened a new log
Aug 06 10:47:25 dist-test-slave-0b58 krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:25.560473 19012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580b03ed60b4a9ebd33046d22e116bb" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42117 } }
I20260806 10:47:25.560868 19012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:25.561002 19012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0580b03ed60b4a9ebd33046d22e116bb, State: Initialized, Role: FOLLOWER
I20260806 10:47:25.561661 19012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580b03ed60b4a9ebd33046d22e116bb" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42117 } }
I20260806 10:47:25.561964 19012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:25.562124 19012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:25.562330 19012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:25.563036 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:25.541000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.16.21.62:42117)
0806 10:47:25.541906 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:25.541935 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:25.541983 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:47:25.543663 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:25.543684 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:25.544033 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:25.544423 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:25.546337 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:25.547563 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:25.549440 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:25.550539 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:25.550673 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:25.555334 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:25.555342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:25.555358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:25.555400 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:25.558572 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:25.559251 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:25.559258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:25.559265 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:25.559438 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:25.559839 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:25.559844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:25.559846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:25.560385 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:25.560692 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:25.561180 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 10:47:25.561594 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":454,"threads_started":1}
I20260806 10:47:25.565313 19012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580b03ed60b4a9ebd33046d22e116bb" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42117 } }
I20260806 10:47:25.565897 19012 leader_election.cc:302] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0580b03ed60b4a9ebd33046d22e116bb; no voters:
I20260806 10:47:25.566520 19012 leader_election.cc:288] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:25.566650 19017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:25.567435 19017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [term 1 LEADER]: Becoming Leader. State: Replica: 0580b03ed60b4a9ebd33046d22e116bb, State: Running, Role: LEADER
I20260806 10:47:25.567968 19017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580b03ed60b4a9ebd33046d22e116bb" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42117 } }
I20260806 10:47:25.568746 19012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:25.571400 19019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0580b03ed60b4a9ebd33046d22e116bb. Latest consensus state: current_term: 1 leader_uuid: "0580b03ed60b4a9ebd33046d22e116bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580b03ed60b4a9ebd33046d22e116bb" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42117 } } }
I20260806 10:47:25.571485 19018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0580b03ed60b4a9ebd33046d22e116bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580b03ed60b4a9ebd33046d22e116bb" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42117 } } }
I20260806 10:47:25.572005 19019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:25.572073 19018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:25.573019 19026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:25.578536 19026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:25.590333 19026 catalog_manager.cc:1347] Generated new cluster ID: 63f7ef37ebf64cb2a6fa77d67525399d
I20260806 10:47:25.590443 19026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:25.604846 19026 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:25.606647 19026 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:25.613654 19026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0580b03ed60b4a9ebd33046d22e116bb: Generated new TSK 0
I20260806 10:47:25.614639 19026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:25.674374 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42117
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:26.008883 19038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:26.009157 19038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:26.009325 19038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:26.016592 19038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:26.016692 19038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:26.016906 19038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:26.026587 19038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:26.027865 19038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:26.029156 19038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:26.042199 19043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:26.042131 19046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:26.042140 19044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:26.042862 19038 server_base.cc:1034] running on GCE node
I20260806 10:47:26.043444 19038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:26.044389 19038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:26.045617 19038 hybrid_clock.cc:648] HybridClock initialized: now 1786013246045569 us; error 56 us; skew 500 ppm
I20260806 10:47:26.045936 19038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:26 dist-test-slave-0b58 krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:26.049086 19038 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:26.050539 19038 webserver.cc:460] Webserver started at http://127.16.21.1:34295/ using document root <none> and password file <none>
I20260806 10:47:26.051247 19038 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:26.051348 19038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:26.051599 19038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:26.054391 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "d7fff531db304b2b848785288292bc77"
format_stamp: "Formatted at 2026-08-06 10:47:26 on dist-test-slave-0b58"
server_key: "5ccb7509f100b2ab1425928f106d07ab"
server_key_iv: "3c558522a1d2e282ba0a04645e5ae555"
server_key_version: "encryptionkey@0"
I20260806 10:47:26.055122 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "d7fff531db304b2b848785288292bc77"
format_stamp: "Formatted at 2026-08-06 10:47:26 on dist-test-slave-0b58"
server_key: "5ccb7509f100b2ab1425928f106d07ab"
server_key_iv: "3c558522a1d2e282ba0a04645e5ae555"
server_key_version: "encryptionkey@0"
I20260806 10:47:26.060178 19038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:47:26.063846 19053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:26.065253 19038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:47:26.065446 19038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "d7fff531db304b2b848785288292bc77"
format_stamp: "Formatted at 2026-08-06 10:47:26 on dist-test-slave-0b58"
server_key: "5ccb7509f100b2ab1425928f106d07ab"
server_key_iv: "3c558522a1d2e282ba0a04645e5ae555"
server_key_version: "encryptionkey@0"
I20260806 10:47:26.065645 19038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:26.090186 19038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:26.094841 19038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:26.095140 19038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:26.096087 19038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:26.097440 19038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:26.097553 19038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:26.097704 19038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:26.097784 19038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:26.131518 19038 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:43985
I20260806 10:47:26.131572 19168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:43985 every 8 connection(s)
I20260806 10:47:26.132877 19038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:26.137369 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19038
I20260806 10:47:26.137599 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:26.138432 16468 external_mini_cluster.cc:1428] Setting key 76e15f23db2a98813e0fb8a53a472d81
Aug 06 10:47:26 dist-test-slave-0b58 krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:26.154069 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:26.135352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:35235 (local address 127.16.21.62:42117)
0806 10:47:26.135814 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:26.135820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:26.136853 (+ 1033us) server_negotiation.cc:407] Connection header received
0806 10:47:26.139395 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:26.139400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:26.139499 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:26.139617 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:26.141188 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.142364 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.143894 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.144628 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.144772 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:26.148015 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:26.148024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:26.148031 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:26.148074 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:26.150783 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:26.151730 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:26.151733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:26.151736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:26.151847 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:26.152337 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:26.152342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:26.152345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:26.152597 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:26.152705 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:26.153652 (+ 947us) server_negotiation.cc:299] Negotiation successful
0806 10:47:26.153819 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":212,"threads_started":1}
I20260806 10:47:26.155608 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:26.135927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:42117 (local address 127.16.21.1:35235)
0806 10:47:26.136675 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:26.136712 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:26.139128 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:26.139820 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:26.139828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:26.140470 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:26.141012 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:26.141030 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.142513 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.142516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:26.143725 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:26.143737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.144833 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.144838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:26.145020 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:26.145706 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:26.145727 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:26.147812 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:26.150921 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:26.150933 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:26.150949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:26.151600 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:26.151952 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:26.151959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:26.151965 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:26.152195 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:26.152731 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:26.152742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:26.153224 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 10:47:26.153530 (+ 306us) client_negotiation.cc:240] Negotiation successful
0806 10:47:26.154065 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":357,"threads_started":1}
I20260806 10:47:26.157349 19169 heartbeater.cc:347] Connected to a master server at 127.16.21.62:42117
I20260806 10:47:26.157692 19169 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:26.158391 19169 heartbeater.cc:511] Master 127.16.21.62:42117 requested a full tablet report, sending...
I20260806 10:47:26.160583 18977 ts_manager.cc:194] Registered new tserver with Master: d7fff531db304b2b848785288292bc77 (127.16.21.1:43985)
I20260806 10:47:26.162272 18977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:35235
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:26.191286 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42117
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:26.515720 19175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:26.516022 19175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:26.516194 19175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:26.523273 19175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:26.523416 19175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:26.523584 19175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:26.533260 19175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:26.534365 19175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:26.535561 19175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:26.548931 19183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:26.548969 19181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:26.548998 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:26.549799 19175 server_base.cc:1034] running on GCE node
I20260806 10:47:26.550386 19175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:26.551270 19175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:26.552498 19175 hybrid_clock.cc:648] HybridClock initialized: now 1786013246552440 us; error 51 us; skew 500 ppm
I20260806 10:47:26.552803 19175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:26 dist-test-slave-0b58 krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:26.555883 19175 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:26.557266 19175 webserver.cc:460] Webserver started at http://127.16.21.2:34867/ using document root <none> and password file <none>
I20260806 10:47:26.557941 19175 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:26.558089 19175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:26.558405 19175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:26.561311 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "c9965879b1f746fa89c02df804a7ca2b"
format_stamp: "Formatted at 2026-08-06 10:47:26 on dist-test-slave-0b58"
server_key: "242d78a4956605d35cd1d8f2c72d7606"
server_key_iv: "4b0cee8409260fc5a1278cfdcdb43395"
server_key_version: "encryptionkey@0"
I20260806 10:47:26.562106 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "c9965879b1f746fa89c02df804a7ca2b"
format_stamp: "Formatted at 2026-08-06 10:47:26 on dist-test-slave-0b58"
server_key: "242d78a4956605d35cd1d8f2c72d7606"
server_key_iv: "4b0cee8409260fc5a1278cfdcdb43395"
server_key_version: "encryptionkey@0"
I20260806 10:47:26.567584 19175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 10:47:26.571425 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:26.572777 19175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:26.572949 19175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "c9965879b1f746fa89c02df804a7ca2b"
format_stamp: "Formatted at 2026-08-06 10:47:26 on dist-test-slave-0b58"
server_key: "242d78a4956605d35cd1d8f2c72d7606"
server_key_iv: "4b0cee8409260fc5a1278cfdcdb43395"
server_key_version: "encryptionkey@0"
I20260806 10:47:26.573173 19175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:26.591010 19175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:26.595319 19175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:26.595608 19175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:26.596542 19175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:26.597851 19175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:26.597957 19175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:26.598122 19175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:26.598197 19175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:26.631763 19175 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:35727
I20260806 10:47:26.631811 19304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:35727 every 8 connection(s)
I20260806 10:47:26.633179 19175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:26.641436 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19175
I20260806 10:47:26.641656 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:26.642221 16468 external_mini_cluster.cc:1428] Setting key 0e07528ebf4c2ff976fbf2d8ed075c2c
Aug 06 10:47:26 dist-test-slave-0b58 krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:26.653124 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:26.635670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45337 (local address 127.16.21.62:42117)
0806 10:47:26.635863 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:26.635867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:26.637067 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 10:47:26.638917 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:26.638921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:26.638992 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:26.639081 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:26.640543 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.641261 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.643327 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.643786 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.643910 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:26.647425 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:26.647430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:26.647433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:26.647464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:26.650098 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:26.650819 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:26.650824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:26.650830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:26.650965 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:26.651404 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:26.651407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:26.651409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:26.651676 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:26.651833 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:26.652546 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 10:47:26.652781 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:47:26.654367 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:26.636109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:42117 (local address 127.16.21.2:45337)
0806 10:47:26.636867 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:26.636909 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:26.638675 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:26.639289 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:26.639297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:26.639800 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:26.640396 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:26.640415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.641426 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.641432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:26.643157 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:26.643169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:26.644034 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:26.644043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:26.644357 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:26.645035 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:26.645062 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:26.647201 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:26.650245 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:26.650254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:26.650269 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:26.650684 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:26.651079 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:26.651084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:26.651088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:26.651277 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:26.651810 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:26.651817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:26.652137 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 10:47:26.652448 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 10:47:26.652968 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":342,"threads_started":1}
I20260806 10:47:26.655910 19305 heartbeater.cc:347] Connected to a master server at 127.16.21.62:42117
I20260806 10:47:26.656214 19305 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:26.657073 19305 heartbeater.cc:511] Master 127.16.21.62:42117 requested a full tablet report, sending...
I20260806 10:47:26.658586 18977 ts_manager.cc:194] Registered new tserver with Master: c9965879b1f746fa89c02df804a7ca2b (127.16.21.2:35727)
I20260806 10:47:26.659828 18977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:45337
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:26.693630 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42117
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:27.015503 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:27.015758 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:27.015914 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:27.022861 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:27.022976 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:27.023170 19310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:27.032536 19310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:27.033794 19310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:27.035009 19310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:27.048358 19315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:27.048460 19318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:27.048389 19316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:27.049188 19310 server_base.cc:1034] running on GCE node
I20260806 10:47:27.049683 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:27.050498 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:27.051702 19310 hybrid_clock.cc:648] HybridClock initialized: now 1786013247051639 us; error 57 us; skew 500 ppm
I20260806 10:47:27.052009 19310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:27 dist-test-slave-0b58 krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013247, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:27.055258 19310 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:27.056700 19310 webserver.cc:460] Webserver started at http://127.16.21.3:43139/ using document root <none> and password file <none>
I20260806 10:47:27.057299 19310 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:27.057396 19310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:27.057643 19310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:27.060398 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "88eeeb5ae68a4fc3ab42250b56b39abe"
format_stamp: "Formatted at 2026-08-06 10:47:27 on dist-test-slave-0b58"
server_key: "a53d90bcc38c015669ec0bd3fa6b67c4"
server_key_iv: "d7e87a2324375e5c060bbd87cc71c61a"
server_key_version: "encryptionkey@0"
I20260806 10:47:27.061141 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "88eeeb5ae68a4fc3ab42250b56b39abe"
format_stamp: "Formatted at 2026-08-06 10:47:27 on dist-test-slave-0b58"
server_key: "a53d90bcc38c015669ec0bd3fa6b67c4"
server_key_iv: "d7e87a2324375e5c060bbd87cc71c61a"
server_key_version: "encryptionkey@0"
I20260806 10:47:27.066124 19310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:27.069772 19325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:27.071087 19310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:27.071264 19310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "88eeeb5ae68a4fc3ab42250b56b39abe"
format_stamp: "Formatted at 2026-08-06 10:47:27 on dist-test-slave-0b58"
server_key: "a53d90bcc38c015669ec0bd3fa6b67c4"
server_key_iv: "d7e87a2324375e5c060bbd87cc71c61a"
server_key_version: "encryptionkey@0"
I20260806 10:47:27.071465 19310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:27.100865 19310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:27.104836 19310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:27.105124 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:27.106072 19310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:27.107396 19310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:27.107496 19310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:27.107640 19310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:27.107719 19310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:27.141111 19310 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:33825
I20260806 10:47:27.141166 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:33825 every 8 connection(s)
I20260806 10:47:27.142330 19310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:27.144199 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19310
I20260806 10:47:27.144425 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:27.144977 16468 external_mini_cluster.cc:1428] Setting key 8f17ba96e9a62b7c43c621f9d0414dee
Aug 06 10:47:27 dist-test-slave-0b58 krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013247, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:27.162523 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:27.145093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:53831 (local address 127.16.21.62:42117)
0806 10:47:27.145320 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:27.145327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:27.146528 (+ 1201us) server_negotiation.cc:407] Connection header received
0806 10:47:27.149114 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:27.149119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:27.149226 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:27.149309 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:27.150696 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:27.151480 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:27.153033 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:27.153425 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:27.153520 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:27.156718 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:27.156723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:27.156726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:27.156755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:27.159647 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:27.160578 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:27.160583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:27.160586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:27.160735 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:27.161104 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:27.161107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:27.161109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:27.161370 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:27.161483 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:27.162090 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 10:47:27.162268 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:47:27.163388 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:27.145502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:42117 (local address 127.16.21.3:53831)
0806 10:47:27.146347 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:27.146405 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:27.148885 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:27.149502 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:27.149510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:27.150020 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:27.150554 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:27.150570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:27.151607 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:27.151610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:27.152912 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:27.152924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:27.153601 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:27.153605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:27.153796 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:27.154385 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:27.154405 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:27.156533 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:27.159800 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:27.159810 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:27.159825 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:27.160434 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:27.160832 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:27.160835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:27.160838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:27.161007 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:27.161475 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:27.161481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:27.161769 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 10:47:27.161970 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 10:47:27.162302 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":354,"threads_started":1}
I20260806 10:47:27.164911 19438 heartbeater.cc:347] Connected to a master server at 127.16.21.62:42117
I20260806 10:47:27.165208 19438 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:27.165380 19169 heartbeater.cc:503] Master 127.16.21.62:42117 was elected leader, sending a full tablet report...
I20260806 10:47:27.166045 19438 heartbeater.cc:511] Master 127.16.21.62:42117 requested a full tablet report, sending...
I20260806 10:47:27.167415 18977 ts_manager.cc:194] Registered new tserver with Master: 88eeeb5ae68a4fc3ab42250b56b39abe (127.16.21.3:33825)
I20260806 10:47:27.168166 18977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:53831
I20260806 10:47:27.172587 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 10:47:27.212622 19171 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:27.210125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.16.21.62:42117)
0806 10:47:27.210317 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:27.210322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:27.210770 (+ 448us) server_negotiation.cc:407] Connection header received
0806 10:47:27.210914 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:27.210918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:27.210994 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:27.211088 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:27.212356 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44254: BlockingRecv error: recv got EOF from 127.0.0.1:44254 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:47:27.216181 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19038
W20260806 10:47:27.219060 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19038: Not authorized: Client connection negotiation failed: client connection to 127.16.21.1:43985: server requires authentication, but client does not have Kerberos credentials available
I20260806 10:47:27.219259 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19038
I20260806 10:47:27.239215 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19175
W20260806 10:47:27.241649 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19175: Not authorized: Client connection negotiation failed: client connection to 127.16.21.2:35727: server requires authentication, but client does not have Kerberos credentials available
I20260806 10:47:27.241858 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19175
I20260806 10:47:27.259578 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19310
W20260806 10:47:27.262305 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19310: Not authorized: Client connection negotiation failed: client connection to 127.16.21.3:33825: server requires authentication, but client does not have Kerberos credentials available
I20260806 10:47:27.262451 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19310
I20260806 10:47:27.278508 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 18944
I20260806 10:47:27.279606 19007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:27.382376 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 18944
2026-08-06T10:47:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3599 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:27 dist-test-slave-0b58 krb5kdc[19456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:27 dist-test-slave-0b58 krb5kdc[19456](info): set up 2 sockets
Aug 06 10:47:27 dist-test-slave-0b58 krb5kdc[19456](info): commencing operation
krb5kdc: starting...
W20260806 10:47:28.492389 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:28 dist-test-slave-0b58 krb5kdc[19456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:28.705942 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35631
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:29.027691 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:29.027971 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:29.028067 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:29.035071 19469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:29.035166 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:29.035218 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:29.035274 19469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:29.035315 19469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:29.044778 19469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:29.045930 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:29.047119 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:29.058761 19475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:29.058787 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:29.058897 19474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:29.059682 19469 server_base.cc:1034] running on GCE node
I20260806 10:47:29.060447 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:29.061717 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:29.062951 19469 hybrid_clock.cc:648] HybridClock initialized: now 1786013249062924 us; error 53 us; skew 500 ppm
I20260806 10:47:29.063282 19469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:29 dist-test-slave-0b58 krb5kdc[19456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:29.070927 19469 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:29.072357 19469 webserver.cc:460] Webserver started at http://127.16.21.62:45701/ using document root <none> and password file <none>
I20260806 10:47:29.073104 19469 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:29.073257 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:29.073560 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:29.076419 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "118e0684a0394f1ea044109a0eae6ab5"
format_stamp: "Formatted at 2026-08-06 10:47:29 on dist-test-slave-0b58"
server_key: "a096a5a806694030dac594117dce1e0e"
server_key_iv: "3e74fe12df087617965a9568451e41d8"
server_key_version: "encryptionkey@0"
I20260806 10:47:29.077208 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "118e0684a0394f1ea044109a0eae6ab5"
format_stamp: "Formatted at 2026-08-06 10:47:29 on dist-test-slave-0b58"
server_key: "a096a5a806694030dac594117dce1e0e"
server_key_iv: "3e74fe12df087617965a9568451e41d8"
server_key_version: "encryptionkey@0"
I20260806 10:47:29.082221 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:47:29.085644 19485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:29.086798 19469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 10:47:29.086979 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "118e0684a0394f1ea044109a0eae6ab5"
format_stamp: "Formatted at 2026-08-06 10:47:29 on dist-test-slave-0b58"
server_key: "a096a5a806694030dac594117dce1e0e"
server_key_iv: "3e74fe12df087617965a9568451e41d8"
server_key_version: "encryptionkey@0"
I20260806 10:47:29.087163 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:29.105965 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:29.109707 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:29.109973 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:29.128854 19469 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35631
I20260806 10:47:29.128928 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35631 every 8 connection(s)
I20260806 10:47:29.130139 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:29.133483 19538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:29.134079 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19469
I20260806 10:47:29.134277 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:29.134912 16468 external_mini_cluster.cc:1428] Setting key 8abc8f822c436a1af0efbe3b57e43424
I20260806 10:47:29.147428 19538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5: Bootstrap starting.
Aug 06 10:47:29 dist-test-slave-0b58 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:29.150934 19538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:29.151968 19538 log.cc:824] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:29.154491 19538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5: No bootstrap required, opened a new log
I20260806 10:47:29.157352 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:29.136773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.16.21.62:35631)
0806 10:47:29.138240 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:29.138251 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:29.138292 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:47:29.139878 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:29.139897 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:29.140233 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:29.140590 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:29.142310 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:29.143528 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:29.145296 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:29.145939 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:29.146062 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:29.150561 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:29.150568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:29.150580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:29.150614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:29.153520 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:29.154168 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:29.154173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:29.154179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:29.154358 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:29.154729 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:29.154732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:29.154734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:29.155215 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:29.155548 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:29.155984 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 10:47:29.156429 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1012,"threads_started":1}
I20260806 10:47:29.161279 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:47:29.163136 19538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118e0684a0394f1ea044109a0eae6ab5" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35631 } }
I20260806 10:47:29.163463 19538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:29.163575 19538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118e0684a0394f1ea044109a0eae6ab5, State: Initialized, Role: FOLLOWER
I20260806 10:47:29.164098 19538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118e0684a0394f1ea044109a0eae6ab5" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35631 } }
I20260806 10:47:29.164350 19538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:29.164486 19538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:29.164666 19538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:29.167063 19538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118e0684a0394f1ea044109a0eae6ab5" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35631 } }
I20260806 10:47:29.167559 19538 leader_election.cc:302] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118e0684a0394f1ea044109a0eae6ab5; no voters:
I20260806 10:47:29.168164 19538 leader_election.cc:288] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:29.168396 19543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:29.170248 19538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:29.172519 19543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 118e0684a0394f1ea044109a0eae6ab5, State: Running, Role: LEADER
I20260806 10:47:29.173197 19543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118e0684a0394f1ea044109a0eae6ab5" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35631 } }
I20260806 10:47:29.175915 19544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118e0684a0394f1ea044109a0eae6ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118e0684a0394f1ea044109a0eae6ab5" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35631 } } }
I20260806 10:47:29.176436 19544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:29.175905 19545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118e0684a0394f1ea044109a0eae6ab5. Latest consensus state: current_term: 1 leader_uuid: "118e0684a0394f1ea044109a0eae6ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118e0684a0394f1ea044109a0eae6ab5" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35631 } } }
I20260806 10:47:29.176913 19545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:29.180397 19556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:29.185393 19556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:29.194442 19556 catalog_manager.cc:1347] Generated new cluster ID: 30e7b2b199ba4107aae92826c32b4886
I20260806 10:47:29.194561 19556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:29.208931 19556 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:29.210716 19556 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:29.214910 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:29.207165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.16.21.62:35631)
0806 10:47:29.207339 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:29.207345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:29.207931 (+ 586us) server_negotiation.cc:407] Connection header received
0806 10:47:29.208231 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:29.208237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:29.208355 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:29.208465 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:29.209639 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:29.210417 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:29.211705 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:29.212172 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:29.212365 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:29.213455 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:29.213460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:29.213463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:29.214052 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:29.214114 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:29.214136 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:29.214265 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:29.214428 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 10:47:29.214552 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":61}
I20260806 10:47:29.220172 19556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 118e0684a0394f1ea044109a0eae6ab5: Generated new TSK 0
I20260806 10:47:29.221118 19556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:47:29.233693 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19469
I20260806 10:47:29.234862 19533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:29.326875 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19469
2026-08-06T10:47:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1939 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:29 dist-test-slave-0b58 krb5kdc[19572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:29 dist-test-slave-0b58 krb5kdc[19572](info): set up 2 sockets
Aug 06 10:47:29 dist-test-slave-0b58 krb5kdc[19572](info): commencing operation
krb5kdc: starting...
W20260806 10:47:30.420028 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:30 dist-test-slave-0b58 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:30.627601 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46113
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:30.950943 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:30.951223 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:30.951314 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:30.958644 19587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:30.958750 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:30.958798 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:30.958859 19587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:30.958902 19587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:30.968811 19587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:30.969964 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:30.971155 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:30.983029 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:30.983176 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:30.983218 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:30.984010 19587 server_base.cc:1034] running on GCE node
I20260806 10:47:30.984790 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:30.986078 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:30.987365 19587 hybrid_clock.cc:648] HybridClock initialized: now 1786013250987322 us; error 48 us; skew 500 ppm
I20260806 10:47:30.987733 19587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:30 dist-test-slave-0b58 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:30.990957 19587 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:30.992373 19587 webserver.cc:460] Webserver started at http://127.16.21.62:46777/ using document root <none> and password file <none>
I20260806 10:47:30.992955 19587 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:30.993055 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:30.993287 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:30.996071 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "5495fa6f249746fba7f8d0ec892934d6"
format_stamp: "Formatted at 2026-08-06 10:47:30 on dist-test-slave-0b58"
server_key: "d7220b2c2b234091b8068a5f336d5552"
server_key_iv: "fe5d71e3399270d5373fd68f51589045"
server_key_version: "encryptionkey@0"
I20260806 10:47:30.996853 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "5495fa6f249746fba7f8d0ec892934d6"
format_stamp: "Formatted at 2026-08-06 10:47:30 on dist-test-slave-0b58"
server_key: "d7220b2c2b234091b8068a5f336d5552"
server_key_iv: "fe5d71e3399270d5373fd68f51589045"
server_key_version: "encryptionkey@0"
I20260806 10:47:31.001811 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:31.005084 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:31.006332 19587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:31.006469 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "5495fa6f249746fba7f8d0ec892934d6"
format_stamp: "Formatted at 2026-08-06 10:47:30 on dist-test-slave-0b58"
server_key: "d7220b2c2b234091b8068a5f336d5552"
server_key_iv: "fe5d71e3399270d5373fd68f51589045"
server_key_version: "encryptionkey@0"
I20260806 10:47:31.006624 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:31.030651 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:31.034353 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:31.034585 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:31.053423 19587 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:46113
I20260806 10:47:31.053453 19653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:46113 every 8 connection(s)
I20260806 10:47:31.054663 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:31.055671 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19587
I20260806 10:47:31.055908 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:31.056823 16468 external_mini_cluster.cc:1428] Setting key fd08210601096abb922ca07519477f78
I20260806 10:47:31.059299 19654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:47:31 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:31.072412 19654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6: Bootstrap starting.
I20260806 10:47:31.075384 19654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:31.076467 19654 log.cc:824] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:31.079281 19654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6: No bootstrap required, opened a new log
I20260806 10:47:31.079705 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:31.058850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.16.21.62:46113)
0806 10:47:31.059659 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:31.059669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:31.059720 (+ 51us) server_negotiation.cc:407] Connection header received
0806 10:47:31.061430 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:31.061450 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:31.061798 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:31.062105 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:31.063790 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:31.065231 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:31.067043 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:31.067722 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:31.067854 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:31.072561 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:31.072574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:31.072591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:31.072639 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:31.075712 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:31.076418 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:31.076431 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:31.076437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:31.076626 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:31.077019 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:31.077024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:31.077031 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:31.077561 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:31.077922 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:31.078326 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 10:47:31.078717 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":386,"threads_started":1}
I20260806 10:47:31.086954 19654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5495fa6f249746fba7f8d0ec892934d6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46113 } }
I20260806 10:47:31.087323 19654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:31.087450 19654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5495fa6f249746fba7f8d0ec892934d6, State: Initialized, Role: FOLLOWER
I20260806 10:47:31.088078 19654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5495fa6f249746fba7f8d0ec892934d6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46113 } }
I20260806 10:47:31.088374 19654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:31.088532 19654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:31.088717 19654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:31.090963 19654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5495fa6f249746fba7f8d0ec892934d6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46113 } }
I20260806 10:47:31.091471 19654 leader_election.cc:302] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5495fa6f249746fba7f8d0ec892934d6; no voters:
I20260806 10:47:31.092052 19654 leader_election.cc:288] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:31.092200 19659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:31.092947 19659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5495fa6f249746fba7f8d0ec892934d6, State: Running, Role: LEADER
I20260806 10:47:31.093549 19659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5495fa6f249746fba7f8d0ec892934d6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46113 } }
I20260806 10:47:31.094130 19654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:31.096081 19661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5495fa6f249746fba7f8d0ec892934d6. Latest consensus state: current_term: 1 leader_uuid: "5495fa6f249746fba7f8d0ec892934d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5495fa6f249746fba7f8d0ec892934d6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46113 } } }
I20260806 10:47:31.096477 19661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:31.096505 19660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5495fa6f249746fba7f8d0ec892934d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5495fa6f249746fba7f8d0ec892934d6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46113 } } }
I20260806 10:47:31.096978 19660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:31.101976 19664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:31.106789 19664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:31.115165 19664 catalog_manager.cc:1347] Generated new cluster ID: d0be79c5d955433598ce9c3b1b984eb0
I20260806 10:47:31.115268 19664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:31.122879 19664 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:31.124082 19664 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:31.129078 19664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5495fa6f249746fba7f8d0ec892934d6: Generated new TSK 0
I20260806 10:47:31.129932 19664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:31.189447 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46113
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:31.528179 19680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:31.528570 19680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:31.528736 19680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:31.535843 19680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:31.535980 19680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:31.536164 19680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:31.545789 19680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:31.547030 19680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:31.548314 19680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:31.561225 19685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:31.561272 19686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:31.561295 19688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:31.562297 19680 server_base.cc:1034] running on GCE node
I20260806 10:47:31.562950 19680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:31.563856 19680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:31.565089 19680 hybrid_clock.cc:648] HybridClock initialized: now 1786013251565052 us; error 54 us; skew 500 ppm
I20260806 10:47:31.565416 19680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:31 dist-test-slave-0b58 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013251, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:31.568696 19680 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:31.570145 19680 webserver.cc:460] Webserver started at http://127.16.21.1:46219/ using document root <none> and password file <none>
I20260806 10:47:31.570817 19680 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:31.570963 19680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:31.571287 19680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:31.574172 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "f46c2c2b94e84e97a0e0ecb4aa7a77dc"
format_stamp: "Formatted at 2026-08-06 10:47:31 on dist-test-slave-0b58"
server_key: "40fca797a07d0ee6367883ee2637166e"
server_key_iv: "432b4813f48baf442e4350c0d1a1b2c6"
server_key_version: "encryptionkey@0"
I20260806 10:47:31.574962 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "f46c2c2b94e84e97a0e0ecb4aa7a77dc"
format_stamp: "Formatted at 2026-08-06 10:47:31 on dist-test-slave-0b58"
server_key: "40fca797a07d0ee6367883ee2637166e"
server_key_iv: "432b4813f48baf442e4350c0d1a1b2c6"
server_key_version: "encryptionkey@0"
I20260806 10:47:31.580224 19680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:47:31.583562 19695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:31.584852 19680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:31.585021 19680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "f46c2c2b94e84e97a0e0ecb4aa7a77dc"
format_stamp: "Formatted at 2026-08-06 10:47:31 on dist-test-slave-0b58"
server_key: "40fca797a07d0ee6367883ee2637166e"
server_key_iv: "432b4813f48baf442e4350c0d1a1b2c6"
server_key_version: "encryptionkey@0"
I20260806 10:47:31.585219 19680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:31.605213 19680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:31.609304 19680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:31.609603 19680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:31.610563 19680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:31.611866 19680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:31.611935 19680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:31.612126 19680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:31.612188 19680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:31.644722 19680 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:45337
I20260806 10:47:31.644850 19809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:45337 every 8 connection(s)
I20260806 10:47:31.646034 19680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:31.652086 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19680
I20260806 10:47:31.652314 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:31.653007 16468 external_mini_cluster.cc:1428] Setting key 6ad68dbd8a5724cc1c52a9c40c1d3c44
Aug 06 10:47:31 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013251, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:31.665762 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:31.648452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:59089 (local address 127.16.21.62:46113)
0806 10:47:31.648874 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:31.648878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:31.649847 (+ 969us) server_negotiation.cc:407] Connection header received
0806 10:47:31.651590 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:31.651594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:31.651660 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:31.651793 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:31.653479 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:31.654377 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:31.656068 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:31.656519 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:31.656661 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:31.660210 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:31.660215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:31.660218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:31.660249 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:31.662733 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:31.663466 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:31.663472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:31.663476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:31.663600 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:31.664020 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:31.664025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:31.664028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:31.664330 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:31.664526 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:31.665335 (+ 809us) server_negotiation.cc:299] Negotiation successful
0806 10:47:31.665500 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":215,"threads_started":1}
I20260806 10:47:31.667290 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:31.649000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46113 (local address 127.16.21.1:59089)
0806 10:47:31.649682 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:31.649730 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:31.651346 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:31.652035 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:31.652042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:31.652694 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:31.653306 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:31.653325 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:31.654530 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:31.654540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:31.655908 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:31.655920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:31.656767 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:31.656775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:31.656997 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:31.657662 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:31.657687 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:31.659994 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:31.662882 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:31.662890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:31.662904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:31.663349 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:31.663711 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:31.663716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:31.663720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:31.663906 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:31.664476 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:31.664483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:31.664910 (+ 427us) client_negotiation.cc:769] Sending connection context
0806 10:47:31.665211 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 10:47:31.665771 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":266,"threads_started":1}
I20260806 10:47:31.668952 19810 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46113
I20260806 10:47:31.669281 19810 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:31.669991 19810 heartbeater.cc:511] Master 127.16.21.62:46113 requested a full tablet report, sending...
I20260806 10:47:31.672003 19619 ts_manager.cc:194] Registered new tserver with Master: f46c2c2b94e84e97a0e0ecb4aa7a77dc (127.16.21.1:45337)
I20260806 10:47:31.673592 19619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:59089
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:31.707427 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46113
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:32.033443 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:32.033737 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:32.033911 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:32.041018 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:32.041164 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:32.041364 19816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:32.051352 19816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:32.052597 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:32.053844 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:32.067190 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:32.067225 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:32.067484 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:32.068576 19816 server_base.cc:1034] running on GCE node
I20260806 10:47:32.069195 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:32.070130 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:32.071353 19816 hybrid_clock.cc:648] HybridClock initialized: now 1786013252071321 us; error 47 us; skew 500 ppm
I20260806 10:47:32.071657 19816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:32.074905 19816 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:32.076303 19816 webserver.cc:460] Webserver started at http://127.16.21.2:33679/ using document root <none> and password file <none>
I20260806 10:47:32.076936 19816 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:32.077085 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:32.077391 19816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:32.080111 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "0b9c809905734ec3947d90f0f458e2b3"
format_stamp: "Formatted at 2026-08-06 10:47:32 on dist-test-slave-0b58"
server_key: "3103ae53690c835c20cc89885e98dcbb"
server_key_iv: "1a639b41ec11f325acd47f57ee107a9d"
server_key_version: "encryptionkey@0"
I20260806 10:47:32.080919 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "0b9c809905734ec3947d90f0f458e2b3"
format_stamp: "Formatted at 2026-08-06 10:47:32 on dist-test-slave-0b58"
server_key: "3103ae53690c835c20cc89885e98dcbb"
server_key_iv: "1a639b41ec11f325acd47f57ee107a9d"
server_key_version: "encryptionkey@0"
I20260806 10:47:32.085844 19816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:32.089308 19831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:32.090647 19816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:47:32.090838 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "0b9c809905734ec3947d90f0f458e2b3"
format_stamp: "Formatted at 2026-08-06 10:47:32 on dist-test-slave-0b58"
server_key: "3103ae53690c835c20cc89885e98dcbb"
server_key_iv: "1a639b41ec11f325acd47f57ee107a9d"
server_key_version: "encryptionkey@0"
I20260806 10:47:32.091032 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:32.114684 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:32.118903 19816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:32.119205 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:32.120154 19816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:32.121486 19816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:32.121591 19816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:32.121735 19816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:32.121814 19816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:32.155303 19816 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:38167
I20260806 10:47:32.155381 19945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:38167 every 8 connection(s)
I20260806 10:47:32.156659 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:32.158427 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19816
I20260806 10:47:32.158598 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:32.159276 16468 external_mini_cluster.cc:1428] Setting key 1b2984794326a9760ae6a3a274b2f691
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:32.178577 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.159461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:39769 (local address 127.16.21.62:46113)
0806 10:47:32.159634 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:32.159639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:32.161141 (+ 1502us) server_negotiation.cc:407] Connection header received
0806 10:47:32.163063 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:32.163069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:32.163152 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:32.163273 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:32.164773 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.165491 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.167723 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.168486 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.168634 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.172516 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:32.172521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:32.172524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:32.172555 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:32.175400 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.176157 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.176166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.176168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.176383 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.176829 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.176834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.176837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.177107 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:32.177303 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:32.177991 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 10:47:32.178236 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:47:32.179654 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.160006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46113 (local address 127.16.21.2:39769)
0806 10:47:32.160968 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:32.161014 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:32.162798 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:32.163497 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:32.163505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:32.163998 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:32.164609 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:32.164627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.165656 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.165661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:32.167555 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:32.167571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.168706 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.168714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:32.169026 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.169779 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:32.169816 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:32.172239 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:32.175568 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:32.175579 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:32.175593 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:32.176023 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:32.176502 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:32.176508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:32.176511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:32.176705 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:32.177255 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:32.177263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:32.177615 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 10:47:32.177882 (+ 267us) client_negotiation.cc:240] Negotiation successful
0806 10:47:32.178406 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":404,"threads_started":1}
I20260806 10:47:32.181308 19946 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46113
I20260806 10:47:32.181635 19946 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:32.182374 19946 heartbeater.cc:511] Master 127.16.21.62:46113 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
I20260806 10:47:32.183857 19619 ts_manager.cc:194] Registered new tserver with Master: 0b9c809905734ec3947d90f0f458e2b3 (127.16.21.2:38167)
I20260806 10:47:32.184724 19619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:39769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:32.215524 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46113
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:32.546900 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:32.547147 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:32.547308 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:32.554419 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:32.554512 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:32.554715 19951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:32.564525 19951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:32.565776 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:32.567108 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:32.580298 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:32.580351 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:32.580375 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:32.581367 19951 server_base.cc:1034] running on GCE node
I20260806 10:47:32.582029 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:32.582954 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:32.584179 19951 hybrid_clock.cc:648] HybridClock initialized: now 1786013252584128 us; error 50 us; skew 500 ppm
I20260806 10:47:32.584590 19951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:32.587723 19951 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:32.589228 19951 webserver.cc:460] Webserver started at http://127.16.21.3:46105/ using document root <none> and password file <none>
I20260806 10:47:32.589886 19951 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:32.589991 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:32.590233 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:32.593104 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "2508bc48798b4f3fbfc2447e707d4eac"
format_stamp: "Formatted at 2026-08-06 10:47:32 on dist-test-slave-0b58"
server_key: "7156e425ecbbfc4b966138e0b710d6e0"
server_key_iv: "00b5dc17ed91bc4d8eea6a5c1c89c325"
server_key_version: "encryptionkey@0"
I20260806 10:47:32.593870 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "2508bc48798b4f3fbfc2447e707d4eac"
format_stamp: "Formatted at 2026-08-06 10:47:32 on dist-test-slave-0b58"
server_key: "7156e425ecbbfc4b966138e0b710d6e0"
server_key_iv: "00b5dc17ed91bc4d8eea6a5c1c89c325"
server_key_version: "encryptionkey@0"
I20260806 10:47:32.599188 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:47:32.602536 19966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:32.603853 19951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:32.604017 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "2508bc48798b4f3fbfc2447e707d4eac"
format_stamp: "Formatted at 2026-08-06 10:47:32 on dist-test-slave-0b58"
server_key: "7156e425ecbbfc4b966138e0b710d6e0"
server_key_iv: "00b5dc17ed91bc4d8eea6a5c1c89c325"
server_key_version: "encryptionkey@0"
I20260806 10:47:32.604213 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:32.637814 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:32.642223 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:32.642545 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:32.643600 19951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:32.645097 19951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:32.645202 19951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:32.645361 19951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:32.645437 19951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:32.676397 19810 heartbeater.cc:503] Master 127.16.21.62:46113 was elected leader, sending a full tablet report...
I20260806 10:47:32.680707 19951 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:42409
I20260806 10:47:32.680759 20080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:42409 every 8 connection(s)
I20260806 10:47:32.682029 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:32.691538 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 19951
I20260806 10:47:32.691706 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:32.692207 16468 external_mini_cluster.cc:1428] Setting key 5b7cce0fc691d661bc4b12ca9d3afcca
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:32.701844 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.684484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:52243 (local address 127.16.21.62:46113)
0806 10:47:32.684940 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:32.684944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:32.685741 (+ 797us) server_negotiation.cc:407] Connection header received
0806 10:47:32.687589 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:32.687595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:32.687678 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:32.687798 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:32.689402 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.690200 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.691753 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.692208 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.692412 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.696550 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:32.696563 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:32.696567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:32.696597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:32.698889 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.699568 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.699573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.699578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.699707 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.700139 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.700144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.700148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.700474 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:32.700626 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:32.701264 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 10:47:32.701474 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":260,"threads_started":1}
I20260806 10:47:32.702759 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.684909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46113 (local address 127.16.21.3:52243)
0806 10:47:32.685585 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:32.685627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:32.687329 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:32.688026 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:32.688033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:32.688669 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:32.689204 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:32.689221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.690354 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.690365 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:32.691585 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:32.691621 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.692501 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.692508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:32.692778 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.693502 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:32.693544 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:32.696362 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:32.699023 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:32.699032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:32.699045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:32.699447 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:32.699830 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:32.699834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:32.699838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:32.700023 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:32.700600 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:32.700607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:32.700907 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 10:47:32.701146 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 10:47:32.701665 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":280,"threads_started":1}
I20260806 10:47:32.703925 20081 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46113
I20260806 10:47:32.704221 20081 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:32.704932 20081 heartbeater.cc:511] Master 127.16.21.62:46113 requested a full tablet report, sending...
I20260806 10:47:32.706468 19619 ts_manager.cc:194] Registered new tserver with Master: 2508bc48798b4f3fbfc2447e707d4eac (127.16.21.3:42409)
I20260806 10:47:32.707474 19619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:52243
I20260806 10:47:32.707888 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:32.763245 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.749789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.16.21.62:46113)
0806 10:47:32.750052 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:32.750058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:32.750267 (+ 209us) server_negotiation.cc:407] Connection header received
0806 10:47:32.750421 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:32.750425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:32.750496 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:32.750584 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:32.751904 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.752699 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.753910 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.754305 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.754396 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.758134 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:32.758139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:32.758142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:32.758175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:32.760862 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.761492 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.761496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.761498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.761616 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.761968 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.761971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.761973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.762303 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:32.762500 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:32.762717 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 10:47:32.762884 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260806 10:47:32.763868 19619 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52766
I20260806 10:47:32.765659 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19680
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:32.781733 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.766790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.16.21.1:45337)
0806 10:47:32.767391 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:32.767397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:32.767421 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:47:32.767475 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:32.767481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:32.767681 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:32.767801 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:32.768992 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.770001 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.771149 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.771688 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.771780 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.776159 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:32.776172 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:32.776191 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:32.776224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:32.778847 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.779484 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.779493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.779496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.779627 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.780020 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.780025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.780029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.780435 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:32.780755 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:32.781094 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 10:47:32.781369 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":295,"threads_started":1}
W20260806 10:47:32.783036 19805 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52154
W20260806 10:47:32.784040 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:32.784314 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19680
I20260806 10:47:32.804431 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19816
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:32.820902 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.805277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.16.21.2:38167)
0806 10:47:32.805841 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:32.805848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:32.805873 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:47:32.805932 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:32.805938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:32.806143 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:32.806296 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:32.807569 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.808758 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.810067 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.810670 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.810811 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.815334 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:32.815345 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:32.815359 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:32.815399 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:32.818140 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.818719 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.818725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.818728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.818856 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.819284 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.819288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.819290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.819651 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:32.819902 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:32.820199 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 10:47:32.820492 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
W20260806 10:47:32.822146 19941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41296
W20260806 10:47:32.823240 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:32.823423 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19816
I20260806 10:47:32.842427 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19951
Aug 06 10:47:32 dist-test-slave-0b58 krb5kdc[19572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:32.858791 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:32.843529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.16.21.3:42409)
0806 10:47:32.844160 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:32.844166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:32.844196 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:47:32.844299 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:32.844311 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:32.844541 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:32.844666 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:32.845951 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.847129 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.848414 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:32.848960 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:32.849055 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:32.853240 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:32.853251 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:32.853265 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:32.853300 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:32.856172 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.856974 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.856982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.856984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.857108 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:32.857430 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:32.857433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:32.857435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:32.857783 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:32.858015 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:32.858293 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 10:47:32.858491 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":331,"threads_started":1}
W20260806 10:47:32.860146 20076 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41260
W20260806 10:47:32.861395 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:47:32.861608 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19951
I20260806 10:47:32.880432 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 19587
I20260806 10:47:32.881676 19649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:32.986563 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 19587
2026-08-06T10:47:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3666 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:33 dist-test-slave-0b58 krb5kdc[20099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:33 dist-test-slave-0b58 krb5kdc[20099](info): set up 2 sockets
Aug 06 10:47:33 dist-test-slave-0b58 krb5kdc[20099](info): commencing operation
krb5kdc: starting...
W20260806 10:47:34.076529 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:34 dist-test-slave-0b58 krb5kdc[20099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:34.299263 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35285
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:34.623596 20112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:34.623907 20112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:34.624075 20112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:34.631342 20112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:34.631479 20112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:34.631577 20112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:34.631675 20112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:34.631752 20112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:34.641396 20112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:34.642632 20112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:34.643923 20112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:34.655646 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:34.655709 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:34.655705 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:34.656580 20112 server_base.cc:1034] running on GCE node
I20260806 10:47:34.657407 20112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:34.658735 20112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:34.659986 20112 hybrid_clock.cc:648] HybridClock initialized: now 1786013254659943 us; error 57 us; skew 500 ppm
I20260806 10:47:34.660393 20112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:34 dist-test-slave-0b58 krb5kdc[20099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013254, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:34.668677 20112 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:34.670203 20112 webserver.cc:460] Webserver started at http://127.16.21.62:41821/ using document root <none> and password file <none>
I20260806 10:47:34.670900 20112 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:34.671052 20112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:34.671380 20112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:34.674461 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "464bffdf65414dd781780e38f0dcbc43"
format_stamp: "Formatted at 2026-08-06 10:47:34 on dist-test-slave-0b58"
server_key: "0d4813ee2d50ad99b858e353e0014343"
server_key_iv: "394ef82b7d3adb8a37e2b781e97bdb93"
server_key_version: "encryptionkey@0"
I20260806 10:47:34.675307 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "464bffdf65414dd781780e38f0dcbc43"
format_stamp: "Formatted at 2026-08-06 10:47:34 on dist-test-slave-0b58"
server_key: "0d4813ee2d50ad99b858e353e0014343"
server_key_iv: "394ef82b7d3adb8a37e2b781e97bdb93"
server_key_version: "encryptionkey@0"
I20260806 10:47:34.680501 20112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:47:34.683806 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:34.685103 20112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:34.685290 20112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "464bffdf65414dd781780e38f0dcbc43"
format_stamp: "Formatted at 2026-08-06 10:47:34 on dist-test-slave-0b58"
server_key: "0d4813ee2d50ad99b858e353e0014343"
server_key_iv: "394ef82b7d3adb8a37e2b781e97bdb93"
server_key_version: "encryptionkey@0"
I20260806 10:47:34.685501 20112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:34.703043 20112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:34.706836 20112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:34.707115 20112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:34.726130 20112 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35285
I20260806 10:47:34.726222 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35285 every 8 connection(s)
I20260806 10:47:34.727483 20112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:34.728543 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20112
I20260806 10:47:34.728782 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:34.729601 16468 external_mini_cluster.cc:1428] Setting key 276239c4077a87b39272c979ca2b6969
I20260806 10:47:34.732194 20180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:47:34 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:34.745518 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43: Bootstrap starting.
I20260806 10:47:34.749627 20180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:34.751080 20180 log.cc:824] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:34.753469 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:34.731654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.16.21.62:35285)
0806 10:47:34.732478 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:34.732489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:34.732531 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:47:34.734149 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:34.734168 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:34.734526 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:34.734834 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:34.736792 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:34.738100 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:34.739890 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:34.740578 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:34.740709 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:34.745266 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:34.745286 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:34.745304 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:34.745347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:34.749237 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:34.749916 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:34.749926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:34.749935 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:34.750178 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:34.750680 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:34.750685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:34.750688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:34.751398 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:34.751738 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:34.752086 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 10:47:34.752467 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":360,"threads_started":1}
I20260806 10:47:34.754092 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43: No bootstrap required, opened a new log
I20260806 10:47:34.761984 20180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464bffdf65414dd781780e38f0dcbc43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35285 } }
I20260806 10:47:34.762360 20180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:34.762468 20180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464bffdf65414dd781780e38f0dcbc43, State: Initialized, Role: FOLLOWER
I20260806 10:47:34.762979 20180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464bffdf65414dd781780e38f0dcbc43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35285 } }
I20260806 10:47:34.763222 20180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:34.763366 20180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:34.763533 20180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:34.765609 20180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464bffdf65414dd781780e38f0dcbc43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35285 } }
I20260806 10:47:34.766084 20180 leader_election.cc:302] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 464bffdf65414dd781780e38f0dcbc43; no voters:
I20260806 10:47:34.766798 20180 leader_election.cc:288] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:34.766901 20185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:34.768118 20185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [term 1 LEADER]: Becoming Leader. State: Replica: 464bffdf65414dd781780e38f0dcbc43, State: Running, Role: LEADER
I20260806 10:47:34.768633 20185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464bffdf65414dd781780e38f0dcbc43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35285 } }
I20260806 10:47:34.768796 20180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:34.771946 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "464bffdf65414dd781780e38f0dcbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464bffdf65414dd781780e38f0dcbc43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35285 } } }
I20260806 10:47:34.771857 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464bffdf65414dd781780e38f0dcbc43. Latest consensus state: current_term: 1 leader_uuid: "464bffdf65414dd781780e38f0dcbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464bffdf65414dd781780e38f0dcbc43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35285 } } }
I20260806 10:47:34.772436 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:34.772413 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:34.774303 20192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:34.779639 20192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:34.788853 20192 catalog_manager.cc:1347] Generated new cluster ID: 57bdf75142d24c2db609c0cc3c794e7b
I20260806 10:47:34.788980 20192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:34.810515 20192 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:34.811827 20192 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:34.823009 20192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464bffdf65414dd781780e38f0dcbc43: Generated new TSK 0
I20260806 10:47:34.823812 20192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:34.887560 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35285
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:35.237524 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:35.237790 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:35.237881 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:35.245141 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:35.245244 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:35.245373 20207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:35.255442 20207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:35.256621 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:35.257911 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:35.271582 20213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:35.271544 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:35.271607 20214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:35.272378 20207 server_base.cc:1034] running on GCE node
I20260806 10:47:35.272948 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:35.273952 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:35.275167 20207 hybrid_clock.cc:648] HybridClock initialized: now 1786013255275123 us; error 58 us; skew 500 ppm
I20260806 10:47:35.275486 20207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:35 dist-test-slave-0b58 krb5kdc[20099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:35.278502 20207 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:35.279917 20207 webserver.cc:460] Webserver started at http://127.16.21.1:45957/ using document root <none> and password file <none>
I20260806 10:47:35.280652 20207 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:35.280797 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:35.281106 20207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:35.283812 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "3baccbdbed424c0099cdf4bd1fb856b3"
format_stamp: "Formatted at 2026-08-06 10:47:35 on dist-test-slave-0b58"
server_key: "945e1b96b134d3d2249e25651112c8d8"
server_key_iv: "4665027981bb58f8169b27b013ca1897"
server_key_version: "encryptionkey@0"
I20260806 10:47:35.284626 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "3baccbdbed424c0099cdf4bd1fb856b3"
format_stamp: "Formatted at 2026-08-06 10:47:35 on dist-test-slave-0b58"
server_key: "945e1b96b134d3d2249e25651112c8d8"
server_key_iv: "4665027981bb58f8169b27b013ca1897"
server_key_version: "encryptionkey@0"
I20260806 10:47:35.289511 20207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 10:47:35.293335 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:35.294513 20207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:35.294682 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "3baccbdbed424c0099cdf4bd1fb856b3"
format_stamp: "Formatted at 2026-08-06 10:47:35 on dist-test-slave-0b58"
server_key: "945e1b96b134d3d2249e25651112c8d8"
server_key_iv: "4665027981bb58f8169b27b013ca1897"
server_key_version: "encryptionkey@0"
I20260806 10:47:35.294870 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:35.311367 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:35.315033 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:35.315317 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:35.316174 20207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:35.317463 20207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:35.317602 20207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:35.317744 20207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:35.317817 20207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:35.350764 20207 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:41371
I20260806 10:47:35.350841 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:41371 every 8 connection(s)
I20260806 10:47:35.352140 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:35.354400 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20207
I20260806 10:47:35.354622 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:35.355486 16468 external_mini_cluster.cc:1428] Setting key be7431bc9b1ef9f80eb40f4f3b38e2f2
Aug 06 10:47:35 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:35.373203 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:35.354807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:46793 (local address 127.16.21.62:35285)
0806 10:47:35.355256 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:35.355260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:35.356212 (+ 952us) server_negotiation.cc:407] Connection header received
0806 10:47:35.358630 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:35.358636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:35.358735 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:35.358855 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:35.360229 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.360948 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.362435 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.362843 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.362976 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:35.366332 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:35.366339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:35.366347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:35.366392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:35.370063 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:35.370724 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:35.370732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:35.370738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:35.370897 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:35.371287 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:35.371295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:35.371301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:35.371663 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:35.371825 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:35.372445 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 10:47:35.372727 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260806 10:47:35.373721 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:35.355199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35285 (local address 127.16.21.1:46793)
0806 10:47:35.356020 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:35.356057 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:35.358373 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:35.359083 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:35.359091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:35.359562 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:35.360064 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:35.360081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.361141 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.361146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:35.362295 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:35.362307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.363065 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.363073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:35.363306 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:35.363954 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:35.363978 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:35.366100 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:35.370194 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:35.370200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:35.370210 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:35.370600 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:35.371005 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:35.371008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:35.371011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:35.371181 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:35.371787 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:35.371792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:35.372061 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 10:47:35.372337 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 10:47:35.372684 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":376,"threads_started":1}
I20260806 10:47:35.375285 20339 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35285
I20260806 10:47:35.375640 20339 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:35.376421 20339 heartbeater.cc:511] Master 127.16.21.62:35285 requested a full tablet report, sending...
I20260806 10:47:35.378461 20144 ts_manager.cc:194] Registered new tserver with Master: 3baccbdbed424c0099cdf4bd1fb856b3 (127.16.21.1:41371)
I20260806 10:47:35.379882 20144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:46793
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:35.409394 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35285
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:35.742203 20345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:35.742518 20345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:35.742662 20345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:35.753013 20345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:35.753114 20345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:35.753232 20345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:35.762420 20345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:35.763477 20345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:35.764727 20345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:35.777297 20350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:35.777380 20351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:35.777381 20353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:35.779047 20345 server_base.cc:1034] running on GCE node
I20260806 10:47:35.779608 20345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:35.780552 20345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:35.781775 20345 hybrid_clock.cc:648] HybridClock initialized: now 1786013255781732 us; error 54 us; skew 500 ppm
I20260806 10:47:35.782083 20345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:35 dist-test-slave-0b58 krb5kdc[20099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:35.785053 20345 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:35.786538 20345 webserver.cc:460] Webserver started at http://127.16.21.2:39321/ using document root <none> and password file <none>
I20260806 10:47:35.787181 20345 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:35.787331 20345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:35.787639 20345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:35.790683 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "cc26017508134543a43e7a28212c926f"
format_stamp: "Formatted at 2026-08-06 10:47:35 on dist-test-slave-0b58"
server_key: "f6148cad25158fd9f3fa36e71312fb48"
server_key_iv: "d59ee9e93e8894537b5a5f14a3f736df"
server_key_version: "encryptionkey@0"
I20260806 10:47:35.791416 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "cc26017508134543a43e7a28212c926f"
format_stamp: "Formatted at 2026-08-06 10:47:35 on dist-test-slave-0b58"
server_key: "f6148cad25158fd9f3fa36e71312fb48"
server_key_iv: "d59ee9e93e8894537b5a5f14a3f736df"
server_key_version: "encryptionkey@0"
I20260806 10:47:35.796376 20345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:47:35.799794 20360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:35.801110 20345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:47:35.801301 20345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "cc26017508134543a43e7a28212c926f"
format_stamp: "Formatted at 2026-08-06 10:47:35 on dist-test-slave-0b58"
server_key: "f6148cad25158fd9f3fa36e71312fb48"
server_key_iv: "d59ee9e93e8894537b5a5f14a3f736df"
server_key_version: "encryptionkey@0"
I20260806 10:47:35.801496 20345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:35.827050 20345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:35.831063 20345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:35.831360 20345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:35.832358 20345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:35.833635 20345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:35.833743 20345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:35.833895 20345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:35.833976 20345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:35.867866 20345 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:40047
I20260806 10:47:35.867949 20474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:40047 every 8 connection(s)
I20260806 10:47:35.869294 20345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:35.869757 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20345
I20260806 10:47:35.869961 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:35.870566 16468 external_mini_cluster.cc:1428] Setting key dc3ea6870f3fa5f3d9d01ccd3938d162
Aug 06 10:47:35 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:35.890241 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:35.872091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:50595 (local address 127.16.21.62:35285)
0806 10:47:35.872332 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:35.872340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:35.873648 (+ 1308us) server_negotiation.cc:407] Connection header received
0806 10:47:35.875751 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:35.875757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:35.875839 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:35.875956 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:35.877428 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.878153 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.879719 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.880178 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.880345 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:35.884246 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:35.884251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:35.884256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:35.884332 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:35.886843 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:35.887759 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:35.887763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:35.887765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:35.887894 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:35.888430 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:35.888434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:35.888436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:35.888711 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:35.888846 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:35.889736 (+ 890us) server_negotiation.cc:299] Negotiation successful
0806 10:47:35.889970 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:47:35.891752 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:35.872723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35285 (local address 127.16.21.2:50595)
0806 10:47:35.873506 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:35.873546 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:35.875425 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:35.876186 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:35.876195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:35.876742 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:35.877267 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:35.877283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.878315 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.878320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:35.879551 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:35.879562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:35.880432 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:35.880440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:35.880672 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:35.881331 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:35.881352 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:35.884021 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:35.886980 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:35.886991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:35.887008 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:35.887643 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:35.888012 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:35.888031 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:35.888037 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:35.888321 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:35.888849 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:35.888859 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:35.889282 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 10:47:35.889617 (+ 335us) client_negotiation.cc:240] Negotiation successful
0806 10:47:35.890265 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":337,"threads_started":1}
I20260806 10:47:35.893262 20475 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35285
I20260806 10:47:35.893563 20475 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:35.894268 20475 heartbeater.cc:511] Master 127.16.21.62:35285 requested a full tablet report, sending...
I20260806 10:47:35.895756 20144 ts_manager.cc:194] Registered new tserver with Master: cc26017508134543a43e7a28212c926f (127.16.21.2:40047)
I20260806 10:47:35.896989 20144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:50595
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:35.927772 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35285
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:36.253374 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:36.253649 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:36.253822 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:36.261220 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:36.261387 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:36.261580 20480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:36.272228 20480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:36.273463 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:36.274838 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:36.287504 20486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:36.287533 20489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:36.287562 20487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:36.288791 20480 server_base.cc:1034] running on GCE node
I20260806 10:47:36.289283 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:36.290218 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:36.291411 20480 hybrid_clock.cc:648] HybridClock initialized: now 1786013256291380 us; error 29 us; skew 500 ppm
I20260806 10:47:36.291689 20480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:36.294734 20480 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:36.296046 20480 webserver.cc:460] Webserver started at http://127.16.21.3:33273/ using document root <none> and password file <none>
I20260806 10:47:36.297006 20480 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:36.297145 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:36.297461 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:36.300490 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "aa4b4ea25ade453b9c09313f89e0f58b"
format_stamp: "Formatted at 2026-08-06 10:47:36 on dist-test-slave-0b58"
server_key: "a2e6cb7e88f34d6346c2eeb0bf709b58"
server_key_iv: "6b1d5f2bf59a22e2394a032d44d67418"
server_key_version: "encryptionkey@0"
I20260806 10:47:36.301234 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "aa4b4ea25ade453b9c09313f89e0f58b"
format_stamp: "Formatted at 2026-08-06 10:47:36 on dist-test-slave-0b58"
server_key: "a2e6cb7e88f34d6346c2eeb0bf709b58"
server_key_iv: "6b1d5f2bf59a22e2394a032d44d67418"
server_key_version: "encryptionkey@0"
I20260806 10:47:36.306169 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:47:36.309494 20496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:36.310700 20480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:36.310873 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "aa4b4ea25ade453b9c09313f89e0f58b"
format_stamp: "Formatted at 2026-08-06 10:47:36 on dist-test-slave-0b58"
server_key: "a2e6cb7e88f34d6346c2eeb0bf709b58"
server_key_iv: "6b1d5f2bf59a22e2394a032d44d67418"
server_key_version: "encryptionkey@0"
I20260806 10:47:36.311065 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:36.324164 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:36.327926 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:36.328189 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:36.329109 20480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:36.330320 20480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:36.330418 20480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:36.330559 20480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:36.330634 20480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:36.364837 20480 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:35451
I20260806 10:47:36.364926 20609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:35451 every 8 connection(s)
I20260806 10:47:36.366016 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:36.367791 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20480
I20260806 10:47:36.368010 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:36.368600 16468 external_mini_cluster.cc:1428] Setting key 88cce154a2d967496ce8c49a955ab172
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:36.382946 20339 heartbeater.cc:503] Master 127.16.21.62:35285 was elected leader, sending a full tablet report...
I20260806 10:47:36.386827 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.368790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:53149 (local address 127.16.21.62:35285)
0806 10:47:36.368972 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:36.368976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:36.370413 (+ 1437us) server_negotiation.cc:407] Connection header received
0806 10:47:36.372551 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:36.372554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:36.372620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:36.372694 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:36.374137 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.374982 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.376594 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.376999 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.377096 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.380597 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:36.380602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:36.380605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:36.380636 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:36.383799 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.384864 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.384868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.384870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.384998 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.385395 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.385398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.385400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.385658 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:36.385788 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:36.386411 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 10:47:36.386577 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:47:36.387825 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.369291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35285 (local address 127.16.21.3:53149)
0806 10:47:36.370230 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:36.370307 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:36.372203 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:36.372906 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:36.372913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:36.373380 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:36.373996 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:36.374013 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.375104 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.375107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:36.376471 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:36.376484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.377204 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.377209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:36.377404 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.378092 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:36.378112 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:36.380412 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:36.383974 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:36.383982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:36.383996 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:36.384627 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:36.385103 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:36.385106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:36.385109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:36.385293 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:36.385778 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:36.385784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:36.386078 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 10:47:36.386303 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 10:47:36.386667 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":374,"threads_started":1}
I20260806 10:47:36.389194 20610 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35285
I20260806 10:47:36.389477 20610 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:36.390208 20610 heartbeater.cc:511] Master 127.16.21.62:35285 requested a full tablet report, sending...
I20260806 10:47:36.391534 20143 ts_manager.cc:194] Registered new tserver with Master: aa4b4ea25ade453b9c09313f89e0f58b (127.16.21.3:35451)
I20260806 10:47:36.392396 20143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:53149
I20260806 10:47:36.396890 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:36.454828 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.439409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.16.21.1:41371)
0806 10:47:36.440730 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:36.440737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:36.440766 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:47:36.440830 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:36.440836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:36.441048 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:36.441200 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:36.442604 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.443646 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.444906 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.445486 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.445586 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.449337 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:36.449348 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:36.449362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:36.449399 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:36.451973 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.452692 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.452699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.452701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.452831 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.453290 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.453293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.453295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.453736 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:36.453957 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:36.454208 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 10:47:36.454431 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":1002,"threads_started":1}
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:36.472069 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.459046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.16.21.62:35285)
0806 10:47:36.459245 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:36.459253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:36.459346 (+ 93us) server_negotiation.cc:407] Connection header received
0806 10:47:36.459525 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:36.459529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:36.459589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:36.459686 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:36.460921 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.461635 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.462973 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.463392 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.463523 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.467303 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:36.467308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:36.467311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:36.467341 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:36.469951 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.470554 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.470557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.470559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.470680 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.470967 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.470970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.470972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.471267 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:36.471437 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:36.471689 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 10:47:36.471840 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 10:47:36.472723 20143 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45818
I20260806 10:47:36.473703 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20207
I20260806 10:47:36.486078 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.474714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.16.21.1:41371)
0806 10:47:36.474967 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:36.474972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:36.475121 (+ 149us) server_negotiation.cc:407] Connection header received
0806 10:47:36.475242 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:36.475245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:36.475305 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:36.475384 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:36.476655 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.477326 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.478478 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.478902 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.478989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.481103 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:36.481108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:36.481111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:36.481140 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:36.483695 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.484242 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.484245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.484247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.484429 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.484748 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.484751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.484753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.484980 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:36.485153 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:36.485479 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 10:47:36.485720 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:47:36.486837 20334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:36.568673 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20207
I20260806 10:47:36.588495 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20345
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:36.603626 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.589468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.16.21.2:40047)
0806 10:47:36.590052 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:36.590058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:36.590083 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:47:36.590143 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:36.590149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:36.590390 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:36.590514 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:36.591645 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.592709 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.593893 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.594428 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.594530 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.598582 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:36.598592 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:36.598606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:36.598634 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:36.601308 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.601849 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.601855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.601857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.601967 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.602254 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.602257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.602259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.602592 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:36.602824 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:36.603059 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 10:47:36.603308 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":329,"threads_started":1}
I20260806 10:47:36.604990 20470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:36.688513 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20345
I20260806 10:47:36.707183 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20480
Aug 06 10:47:36 dist-test-slave-0b58 krb5kdc[20099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:36.722627 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:36.708165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.16.21.3:35451)
0806 10:47:36.708878 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:36.708884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:36.708910 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:47:36.708969 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:36.708975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:36.709199 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:36.709315 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:36.710452 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.711370 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.712665 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:36.713202 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:36.713310 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:36.717577 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:36.717588 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:36.717601 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:36.717631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:36.720190 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.720881 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.720888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.720890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.721032 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:36.721364 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:36.721367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:36.721369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:36.721678 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:36.721912 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:36.722155 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 10:47:36.722354 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":409,"threads_started":1}
I20260806 10:47:36.723850 20605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:36.806064 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20480
I20260806 10:47:36.827057 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20112
I20260806 10:47:36.828334 20175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:36.933445 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20112
2026-08-06T10:47:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3946 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:37 dist-test-slave-0b58 krb5kdc[20629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:37 dist-test-slave-0b58 krb5kdc[20629](info): set up 2 sockets
Aug 06 10:47:37 dist-test-slave-0b58 krb5kdc[20629](info): commencing operation
krb5kdc: starting...
W20260806 10:47:38.068500 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:38 dist-test-slave-0b58 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:38.295552 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:34171
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:38.616590 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:38.616906 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:38.617082 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:38.624032 20642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:38.624182 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:38.624318 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:38.624431 20642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:38.624526 20642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:38.633901 20642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:38.635116 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:38.636397 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:38.647445 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:38.647404 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:38.647503 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:38.648298 20642 server_base.cc:1034] running on GCE node
I20260806 10:47:38.649020 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:38.650421 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:38.651657 20642 hybrid_clock.cc:648] HybridClock initialized: now 1786013258651610 us; error 35 us; skew 500 ppm
I20260806 10:47:38.652001 20642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:38 dist-test-slave-0b58 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013258, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:38.655030 20642 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:38.656203 20642 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:38.656378 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:38.656709 20642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:38.659669 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "6aaedaf047744f6aa1f5da11e8950648"
format_stamp: "Formatted at 2026-08-06 10:47:38 on dist-test-slave-0b58"
server_key: "60ded5c30dc91dd1a167f1045520c267"
server_key_iv: "3ce7ec0d1987889b2cbf77b8def371db"
server_key_version: "encryptionkey@0"
I20260806 10:47:38.660517 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "6aaedaf047744f6aa1f5da11e8950648"
format_stamp: "Formatted at 2026-08-06 10:47:38 on dist-test-slave-0b58"
server_key: "60ded5c30dc91dd1a167f1045520c267"
server_key_iv: "3ce7ec0d1987889b2cbf77b8def371db"
server_key_version: "encryptionkey@0"
I20260806 10:47:38.665257 20642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:47:38.668604 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:38.669814 20642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:47:38.670007 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "6aaedaf047744f6aa1f5da11e8950648"
format_stamp: "Formatted at 2026-08-06 10:47:38 on dist-test-slave-0b58"
server_key: "60ded5c30dc91dd1a167f1045520c267"
server_key_iv: "3ce7ec0d1987889b2cbf77b8def371db"
server_key_version: "encryptionkey@0"
I20260806 10:47:38.670228 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:38.694959 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:38.698765 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:38.699030 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:38.717829 20642 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:34171
I20260806 10:47:38.717861 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:34171 every 8 connection(s)
I20260806 10:47:38.718704 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:38.722069 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:38.726190 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20642
I20260806 10:47:38.726430 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:38.727223 16468 external_mini_cluster.cc:1428] Setting key 4af4ffe927e337fb8b4ddb2e7f0ae84d
I20260806 10:47:38.733652 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: Bootstrap starting.
I20260806 10:47:38.737099 20708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:38.738252 20708 log.cc:824] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:38.740857 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: No bootstrap required, opened a new log
Aug 06 10:47:38 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:38.751075 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:38.729293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.16.21.62:34171)
0806 10:47:38.730060 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:38.730070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:38.730113 (+ 43us) server_negotiation.cc:407] Connection header received
0806 10:47:38.732085 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:38.732106 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:38.732537 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:38.732824 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:38.734758 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:38.736096 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:38.737893 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:38.738592 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:38.738778 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:38.742765 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:38.742783 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:38.742805 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:38.742857 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:38.746809 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:38.747441 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:38.747450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:38.747457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:38.747669 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:38.748028 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:38.748033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:38.748036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:38.748727 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:38.749053 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:38.749348 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 10:47:38.749811 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":387,"threads_started":1}
I20260806 10:47:38.751539 20708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaedaf047744f6aa1f5da11e8950648" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34171 } }
I20260806 10:47:38.751995 20708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:38.752106 20708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaedaf047744f6aa1f5da11e8950648, State: Initialized, Role: FOLLOWER
I20260806 10:47:38.752717 20708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaedaf047744f6aa1f5da11e8950648" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34171 } }
I20260806 10:47:38.752951 20708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:38.753091 20708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:38.753290 20708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:38.755692 20708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaedaf047744f6aa1f5da11e8950648" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34171 } }
I20260806 10:47:38.756162 20708 leader_election.cc:302] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aaedaf047744f6aa1f5da11e8950648; no voters:
I20260806 10:47:38.756858 20708 leader_election.cc:288] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:38.756990 20713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:38.758023 20713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [term 1 LEADER]: Becoming Leader. State: Replica: 6aaedaf047744f6aa1f5da11e8950648, State: Running, Role: LEADER
I20260806 10:47:38.758666 20713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaedaf047744f6aa1f5da11e8950648" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34171 } }
I20260806 10:47:38.759332 20708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:38.765921 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aaedaf047744f6aa1f5da11e8950648. Latest consensus state: current_term: 1 leader_uuid: "6aaedaf047744f6aa1f5da11e8950648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaedaf047744f6aa1f5da11e8950648" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34171 } } }
I20260806 10:47:38.766304 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:38.766778 20714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aaedaf047744f6aa1f5da11e8950648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaedaf047744f6aa1f5da11e8950648" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34171 } } }
I20260806 10:47:38.767094 20714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648 [sys.catalog]: This master's current role is: LEADER
W20260806 10:47:38.776923 20728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:47:38.777055 20728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:47:38.777252 20729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:38.779493 20729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:38.787899 20729 catalog_manager.cc:1347] Generated new cluster ID: 76eb1eea8c7f4b60a8971820046f0ce5
I20260806 10:47:38.787999 20729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:38.825737 20729 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:38.827294 20729 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:38.839241 20729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: Generated new TSK 0
I20260806 10:47:38.839990 20729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:38.911919 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:34171
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:39.252982 20735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:39.253207 20735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:39.253295 20735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:39.260427 20735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:39.260531 20735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:39.260703 20735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:39.270078 20735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:39.271327 20735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:39.272893 20735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:39.284960 20740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:39.286571 20741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:39.286594 20743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:39.287039 20735 server_base.cc:1034] running on GCE node
I20260806 10:47:39.287616 20735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:39.288589 20735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:39.289836 20735 hybrid_clock.cc:648] HybridClock initialized: now 1786013259289792 us; error 55 us; skew 500 ppm
I20260806 10:47:39.290143 20735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:39 dist-test-slave-0b58 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:39.293267 20735 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:39.294381 20735 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:39.294516 20735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:39.294839 20735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:39.297721 20735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "36fbce56ead24df1b6eb307b25a9bdf4"
format_stamp: "Formatted at 2026-08-06 10:47:39 on dist-test-slave-0b58"
server_key: "14a1480b82c1b4b5d809820ce82e1ddf"
server_key_iv: "409281912b9f04a58f0520e52536de7a"
server_key_version: "encryptionkey@0"
I20260806 10:47:39.298533 20735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "36fbce56ead24df1b6eb307b25a9bdf4"
format_stamp: "Formatted at 2026-08-06 10:47:39 on dist-test-slave-0b58"
server_key: "14a1480b82c1b4b5d809820ce82e1ddf"
server_key_iv: "409281912b9f04a58f0520e52536de7a"
server_key_version: "encryptionkey@0"
I20260806 10:47:39.303311 20735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:47:39.306699 20750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:39.307982 20735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:39.308147 20735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "36fbce56ead24df1b6eb307b25a9bdf4"
format_stamp: "Formatted at 2026-08-06 10:47:39 on dist-test-slave-0b58"
server_key: "14a1480b82c1b4b5d809820ce82e1ddf"
server_key_iv: "409281912b9f04a58f0520e52536de7a"
server_key_version: "encryptionkey@0"
I20260806 10:47:39.308441 20735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:39.354820 20735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:39.358711 20735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:39.359059 20735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:39.360179 20735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:39.361728 20735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:39.361879 20735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 10:47:39.362027 20735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:39.362103 20735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:39.395718 20735 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:39005
I20260806 10:47:39.395803 20863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:39005 every 8 connection(s)
I20260806 10:47:39.396633 20735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:39.398433 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20735
I20260806 10:47:39.398684 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:39.399506 16468 external_mini_cluster.cc:1428] Setting key 3e8b6221a8eb9e9ff223a826c20437f5
Aug 06 10:47:39 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:39.419018 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:39.398929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:51745 (local address 127.16.21.62:34171)
0806 10:47:39.399458 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:39.399464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:39.400548 (+ 1084us) server_negotiation.cc:407] Connection header received
0806 10:47:39.402935 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:39.402939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:39.403000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:39.403089 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:39.405141 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.405850 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.408335 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.408719 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.408813 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:39.413367 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:39.413372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:39.413375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:39.413405 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:39.415895 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:39.416707 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:39.416711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:39.416713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:39.416828 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:39.417303 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:39.417312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:39.417314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:39.417574 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:39.417740 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:39.418423 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 10:47:39.418653 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
I20260806 10:47:39.419874 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:39.399475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34171 (local address 127.16.21.1:51745)
0806 10:47:39.400373 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:39.400417 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:39.402668 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:39.403342 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:39.403355 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:39.404150 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:39.404968 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:39.405000 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.406006 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.406014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:39.408146 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:39.408164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.408937 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.408946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:39.409257 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:39.409957 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:39.409989 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:39.413112 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:39.416045 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:39.416053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:39.416066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:39.416569 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:39.416943 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:39.416948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:39.416952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:39.417169 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:39.417709 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:39.417716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:39.418039 (+ 323us) client_negotiation.cc:769] Sending connection context
0806 10:47:39.418292 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 10:47:39.418754 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":325,"threads_started":1}
I20260806 10:47:39.421561 20864 heartbeater.cc:347] Connected to a master server at 127.16.21.62:34171
I20260806 10:47:39.421919 20864 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:39.422745 20864 heartbeater.cc:511] Master 127.16.21.62:34171 requested a full tablet report, sending...
I20260806 10:47:39.425194 20673 ts_manager.cc:194] Registered new tserver with Master: 36fbce56ead24df1b6eb307b25a9bdf4 (127.16.21.1:39005)
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
I20260806 10:47:39.426838 20673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:51745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:39.453552 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:34171
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:39.772944 20870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:39.773196 20870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:39.773291 20870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:39.780508 20870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:39.780609 20870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:39.780720 20870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:39.790364 20870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:39.791635 20870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:39.792868 20870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:39.805922 20875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:39.806000 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:39.806034 20876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:39.807046 20870 server_base.cc:1034] running on GCE node
I20260806 10:47:39.807642 20870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:39.808593 20870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:39.809820 20870 hybrid_clock.cc:648] HybridClock initialized: now 1786013259809784 us; error 45 us; skew 500 ppm
I20260806 10:47:39.810134 20870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:39 dist-test-slave-0b58 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:39.817941 20870 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:39.819048 20870 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:39.819185 20870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:39.819506 20870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:39.822530 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "638c1c5325c94ad5ac6370c1d3665622"
format_stamp: "Formatted at 2026-08-06 10:47:39 on dist-test-slave-0b58"
server_key: "dd14944a96acbb862fbc6e2f31317609"
server_key_iv: "e1fa212f74bdcbc04c475ca9221f99ee"
server_key_version: "encryptionkey@0"
I20260806 10:47:39.823344 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "638c1c5325c94ad5ac6370c1d3665622"
format_stamp: "Formatted at 2026-08-06 10:47:39 on dist-test-slave-0b58"
server_key: "dd14944a96acbb862fbc6e2f31317609"
server_key_iv: "e1fa212f74bdcbc04c475ca9221f99ee"
server_key_version: "encryptionkey@0"
I20260806 10:47:39.828190 20870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 10:47:39.831902 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:39.833515 20870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:47:39.833689 20870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "638c1c5325c94ad5ac6370c1d3665622"
format_stamp: "Formatted at 2026-08-06 10:47:39 on dist-test-slave-0b58"
server_key: "dd14944a96acbb862fbc6e2f31317609"
server_key_iv: "e1fa212f74bdcbc04c475ca9221f99ee"
server_key_version: "encryptionkey@0"
I20260806 10:47:39.833913 20870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:39.863984 20870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:39.868323 20870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:39.868599 20870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:39.869578 20870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:39.870913 20870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:39.871025 20870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:39.871152 20870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:39.871248 20870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:39.904803 20870 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:45143
I20260806 10:47:39.904888 20998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:45143 every 8 connection(s)
I20260806 10:47:39.905645 20870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:39.913038 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 20870
I20260806 10:47:39.913285 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:39.913820 16468 external_mini_cluster.cc:1428] Setting key f73ebe60bc8691ac059644051b1b5c23
Aug 06 10:47:39 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013259, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:39.927433 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:39.908198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:34005 (local address 127.16.21.62:34171)
0806 10:47:39.908419 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:39.908425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:39.909641 (+ 1216us) server_negotiation.cc:407] Connection header received
0806 10:47:39.911389 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:39.911393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:39.911487 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:39.911611 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:39.913328 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.914358 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.916554 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.917365 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.917536 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:39.920928 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:39.920936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:39.920943 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:39.920986 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:39.924468 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:39.925165 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:39.925172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:39.925178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:39.925336 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:39.925719 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:39.925725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:39.925731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:39.926030 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:39.926137 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:39.927042 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 10:47:39.927209 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:47:39.928915 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:39.908745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34171 (local address 127.16.21.2:34005)
0806 10:47:39.909486 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:39.909526 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:39.911111 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:39.911815 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:39.911824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:39.912488 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:39.913103 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:39.913130 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.914551 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.914556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:39.916402 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:39.916415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:39.917576 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:39.917581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:39.917761 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:39.918453 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:39.918479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:39.920653 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:39.924607 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:39.924613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:39.924623 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:39.925030 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:39.925448 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:39.925451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:39.925453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:39.925609 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:39.926169 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:39.926179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:39.926625 (+ 446us) client_negotiation.cc:769] Sending connection context
0806 10:47:39.926966 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 10:47:39.927506 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":290,"threads_started":1}
I20260806 10:47:39.930354 20999 heartbeater.cc:347] Connected to a master server at 127.16.21.62:34171
I20260806 10:47:39.930644 20999 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:39.931233 20999 heartbeater.cc:511] Master 127.16.21.62:34171 requested a full tablet report, sending...
I20260806 10:47:39.932657 20673 ts_manager.cc:194] Registered new tserver with Master: 638c1c5325c94ad5ac6370c1d3665622 (127.16.21.2:45143)
I20260806 10:47:39.933501 20673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:34005
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:39.967518 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:34171
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:40.296032 21004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:40.296353 21004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:40.296542 21004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:40.303395 21004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:40.303531 21004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:40.303727 21004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:40.313277 21004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:40.314461 21004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:40.315681 21004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:40.327916 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:40.328012 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:40.328040 21012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:40.329222 21004 server_base.cc:1034] running on GCE node
I20260806 10:47:40.329785 21004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:40.330575 21004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:40.331775 21004 hybrid_clock.cc:648] HybridClock initialized: now 1786013260331719 us; error 58 us; skew 500 ppm
I20260806 10:47:40.332037 21004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:40 dist-test-slave-0b58 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013260, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:40.335119 21004 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:40.336135 21004 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:40.336220 21004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:40.336576 21004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:40.339617 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "1cac1e5fe35a44098f1c31a1d03b3968"
format_stamp: "Formatted at 2026-08-06 10:47:40 on dist-test-slave-0b58"
server_key: "4a5a21282e5cfbc0b20ee8f53e4abe81"
server_key_iv: "1d00cd50e064ceef4b04b5aeab3de68e"
server_key_version: "encryptionkey@0"
I20260806 10:47:40.340457 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "1cac1e5fe35a44098f1c31a1d03b3968"
format_stamp: "Formatted at 2026-08-06 10:47:40 on dist-test-slave-0b58"
server_key: "4a5a21282e5cfbc0b20ee8f53e4abe81"
server_key_iv: "1d00cd50e064ceef4b04b5aeab3de68e"
server_key_version: "encryptionkey@0"
I20260806 10:47:40.345183 21004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:47:40.348358 21018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:40.349552 21004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 10:47:40.349718 21004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "1cac1e5fe35a44098f1c31a1d03b3968"
format_stamp: "Formatted at 2026-08-06 10:47:40 on dist-test-slave-0b58"
server_key: "4a5a21282e5cfbc0b20ee8f53e4abe81"
server_key_iv: "1d00cd50e064ceef4b04b5aeab3de68e"
server_key_version: "encryptionkey@0"
I20260806 10:47:40.349927 21004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:40.371592 21004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:40.375430 21004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:40.375633 21004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:40.376482 21004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:40.377714 21004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:40.377807 21004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:40.377889 21004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:40.377941 21004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:40.410113 21004 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:32857
I20260806 10:47:40.410209 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:32857 every 8 connection(s)
I20260806 10:47:40.410876 21004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:40.418437 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21004
I20260806 10:47:40.418635 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:40.419158 16468 external_mini_cluster.cc:1428] Setting key 60700b020476d1ea9824c2df146094ab
Aug 06 10:47:40 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013260, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:40.429731 20864 heartbeater.cc:503] Master 127.16.21.62:34171 was elected leader, sending a full tablet report...
I20260806 10:47:40.430292 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.413376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:58835 (local address 127.16.21.62:34171)
0806 10:47:40.413581 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.413587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.414724 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 10:47:40.416543 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.416548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.416638 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.416725 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:40.418123 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.419263 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.420941 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.421363 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.421457 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.424871 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:40.424876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:40.424879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:40.424908 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:40.427557 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.428173 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.428177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.428179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.428354 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.428710 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.428713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.428715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.428976 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:40.429083 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.429724 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.429947 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:47:40.431337 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.413874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34171 (local address 127.16.21.3:58835)
0806 10:47:40.414550 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:40.414586 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:40.416216 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:40.416910 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:40.416918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:40.417405 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:40.417914 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.417930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.419426 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.419430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.420820 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.420832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.421580 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.421591 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.421860 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.422555 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:40.422575 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:40.424699 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:40.427669 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:40.427674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:40.427684 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:40.428068 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:40.428448 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:40.428451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:40.428453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:40.428612 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:40.429077 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:40.429082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:40.429355 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 10:47:40.429594 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 10:47:40.430077 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":284,"threads_started":1}
I20260806 10:47:40.432646 21133 heartbeater.cc:347] Connected to a master server at 127.16.21.62:34171
I20260806 10:47:40.432940 21133 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:40.433606 21133 heartbeater.cc:511] Master 127.16.21.62:34171 requested a full tablet report, sending...
I20260806 10:47:40.434906 20673 ts_manager.cc:194] Registered new tserver with Master: 1cac1e5fe35a44098f1c31a1d03b3968 (127.16.21.3:32857)
I20260806 10:47:40.435583 20673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:58835
I20260806 10:47:40.446273 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:47:40.461985 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.450378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.16.21.62:34171)
0806 10:47:40.450569 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.450576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.451017 (+ 441us) server_negotiation.cc:407] Connection header received
0806 10:47:40.451130 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.451134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.451188 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.451261 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:40.452604 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.453327 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.454500 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.454964 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.455054 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.457243 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:40.457247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:40.457250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:40.457277 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:40.459650 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.460184 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.460187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.460189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.460376 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.460674 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.460677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.460679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.461019 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:40.461240 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.461477 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.461609 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:47:40.465502 20673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:40.472074 20673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:40.509335 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.484750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.16.21.1:39005)
0806 10:47:40.501571 (+ 16821us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.501576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.501598 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:47:40.501669 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.501673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.501833 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.502045 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:40.502598 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.504657 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.506892 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.508069 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.508225 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.508505 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.508782 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.508951 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16677,"thread_start_us":16497,"threads_started":1}
I20260806 10:47:40.509416 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.484031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:39005 (local address 127.0.0.1:42382)
0806 10:47:40.500317 (+ 16286us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:40.500334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:40.500437 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:40.502021 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:40.502025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:40.502051 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:40.502414 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.502430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.504793 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.504800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.506489 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.506502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.508315 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.508323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.508518 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.508586 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 10:47:40.508740 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 10:47:40.508947 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16125,"mutex_wait_us":96,"thread_start_us":15800,"threads_started":1}
I20260806 10:47:40.509716 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.483551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.16.21.2:45143)
0806 10:47:40.498808 (+ 15257us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.498820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.500231 (+ 1411us) server_negotiation.cc:407] Connection header received
0806 10:47:40.500370 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.500379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.500610 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.500802 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:40.502061 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.503420 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.506677 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.508139 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.508288 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.508476 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.509256 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.509437 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15053,"thread_start_us":16471,"threads_started":1}
I20260806 10:47:40.510087 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.483843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:32857 (local address 127.0.0.1:55032)
0806 10:47:40.500106 (+ 16263us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:40.500127 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:40.500275 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:40.501133 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:40.501140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:40.501180 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:40.501795 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.501823 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.503753 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.503764 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.506480 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.506496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.508715 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.508721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.508955 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.509052 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 10:47:40.509141 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 10:47:40.509290 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16130,"mutex_wait_us":268,"spinlock_wait_cycles":768,"thread_start_us":16163,"threads_started":1}
I20260806 10:47:40.510094 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.484236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.16.21.3:32857)
0806 10:47:40.499460 (+ 15224us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.499468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.500405 (+ 937us) server_negotiation.cc:407] Connection header received
0806 10:47:40.500489 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.500494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.500658 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.500802 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:40.502024 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.503613 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.506677 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.508139 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.508286 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.508514 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.509507 (+ 993us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.509658 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15073,"thread_start_us":15195,"threads_started":1}
I20260806 10:47:40.510661 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.483527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45143 (local address 127.0.0.1:54138)
0806 10:47:40.499420 (+ 15893us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:40.499473 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:40.499667 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:40.501133 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:40.501140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:40.501186 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:40.501856 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.501869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.503530 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.503537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.506481 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.506496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.508671 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.508679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.508949 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.509050 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 10:47:40.509140 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 10:47:40.510286 (+ 1146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15603,"mutex_wait_us":1189,"spinlock_wait_cycles":4864,"thread_start_us":15465,"threads_started":1}
I20260806 10:47:40.513705 20799 tablet_service.cc:1467] Processing CreateTablet for tablet 7c66206af88848ef991b8be6548487a6 (DEFAULT_TABLE table=test-table [id=d006b8ed57f54edaa02400cd62f67839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:40.513764 21067 tablet_service.cc:1467] Processing CreateTablet for tablet 7c66206af88848ef991b8be6548487a6 (DEFAULT_TABLE table=test-table [id=d006b8ed57f54edaa02400cd62f67839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:40.514987 20799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c66206af88848ef991b8be6548487a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:40.515106 21067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c66206af88848ef991b8be6548487a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:40.515471 20932 tablet_service.cc:1467] Processing CreateTablet for tablet 7c66206af88848ef991b8be6548487a6 (DEFAULT_TABLE table=test-table [id=d006b8ed57f54edaa02400cd62f67839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:40.516750 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c66206af88848ef991b8be6548487a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:40.525810 21152 tablet_bootstrap.cc:492] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Bootstrap starting.
I20260806 10:47:40.529254 21154 tablet_bootstrap.cc:492] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Bootstrap starting.
I20260806 10:47:40.530318 21152 tablet_bootstrap.cc:654] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:40.531169 21155 tablet_bootstrap.cc:492] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Bootstrap starting.
I20260806 10:47:40.531998 21152 log.cc:824] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:40.532732 21154 tablet_bootstrap.cc:654] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:40.534070 21154 log.cc:824] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:40.534473 21155 tablet_bootstrap.cc:654] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:40.535423 21155 log.cc:824] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:40.535459 21152 tablet_bootstrap.cc:492] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: No bootstrap required, opened a new log
I20260806 10:47:40.535759 21152 ts_tablet_manager.cc:1397] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260806 10:47:40.538199 21155 tablet_bootstrap.cc:492] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: No bootstrap required, opened a new log
I20260806 10:47:40.538502 21155 ts_tablet_manager.cc:1397] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260806 10:47:40.539654 21154 tablet_bootstrap.cc:492] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: No bootstrap required, opened a new log
I20260806 10:47:40.539913 21154 ts_tablet_manager.cc:1397] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 10:47:40.543846 21152 raft_consensus.cc:348] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.544194 21152 raft_consensus.cc:374] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:40.544359 21152 raft_consensus.cc:729] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fbce56ead24df1b6eb307b25a9bdf4, State: Initialized, Role: FOLLOWER
I20260806 10:47:40.544999 21152 consensus_queue.cc:260] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.546784 21152 ts_tablet_manager.cc:1428] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 10:47:40.549074 21155 raft_consensus.cc:348] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.549582 21155 raft_consensus.cc:374] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:40.549714 21155 raft_consensus.cc:729] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638c1c5325c94ad5ac6370c1d3665622, State: Initialized, Role: FOLLOWER
I20260806 10:47:40.550344 21155 consensus_queue.cc:260] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.550410 21154 raft_consensus.cc:348] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.550837 21154 raft_consensus.cc:374] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:40.550994 21154 raft_consensus.cc:729] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cac1e5fe35a44098f1c31a1d03b3968, State: Initialized, Role: FOLLOWER
I20260806 10:47:40.551549 20999 heartbeater.cc:503] Master 127.16.21.62:34171 was elected leader, sending a full tablet report...
I20260806 10:47:40.551662 21154 consensus_queue.cc:260] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.552217 21155 ts_tablet_manager.cc:1428] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260806 10:47:40.552783 21133 heartbeater.cc:503] Master 127.16.21.62:34171 was elected leader, sending a full tablet report...
I20260806 10:47:40.553347 21154 ts_tablet_manager.cc:1428] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260806 10:47:40.648854 20865 tablet.cc:2369] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:40.657727 21000 tablet.cc:2369] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:40.662421 21134 tablet.cc:2369] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:40.690655 21159 raft_consensus.cc:482] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:40.690936 21159 raft_consensus.cc:504] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.692783 21159 leader_election.cc:288] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 638c1c5325c94ad5ac6370c1d3665622 (127.16.21.2:45143), 1cac1e5fe35a44098f1c31a1d03b3968 (127.16.21.3:32857)
I20260806 10:47:40.699792 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.693734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:39069 (local address 127.16.21.3:32857)
0806 10:47:40.694005 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.694013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.694481 (+ 468us) server_negotiation.cc:407] Connection header received
0806 10:47:40.694713 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.694717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.694779 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.694878 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:40.695487 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.696229 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.698062 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.698845 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.698995 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.699100 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.699413 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.699536 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 10:47:40.699867 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.693150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:32857 (local address 127.16.21.1:39069)
0806 10:47:40.694374 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:40.694395 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:40.694557 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:40.694907 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:40.694911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:40.694943 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:40.695343 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.695352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.696418 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.696423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.697911 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.697923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.699048 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.699054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.699249 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.699274 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 10:47:40.699375 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:47:40.699499 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":862,"threads_started":1}
I20260806 10:47:40.699966 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.693452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:46897 (local address 127.16.21.2:45143)
0806 10:47:40.693721 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.693730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.694662 (+ 932us) server_negotiation.cc:407] Connection header received
0806 10:47:40.694872 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.694876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.694934 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.695042 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:40.695707 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.696470 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.698247 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.699006 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.699172 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.699274 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.699569 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.699706 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 10:47:40.700346 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.693161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45143 (local address 127.16.21.1:46897)
0806 10:47:40.694545 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:40.694558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:40.694710 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:40.695065 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:40.695069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:40.695110 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:40.695513 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.695522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.696613 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.696621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.698072 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:40.698083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.699257 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.699263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:40.699409 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.699443 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:47:40.699542 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:47:40.699654 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":4,"thread_start_us":1039,"threads_started":1}
I20260806 10:47:40.701083 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c66206af88848ef991b8be6548487a6" candidate_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" is_pre_election: true
I20260806 10:47:40.701326 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c66206af88848ef991b8be6548487a6" candidate_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638c1c5325c94ad5ac6370c1d3665622" is_pre_election: true
I20260806 10:47:40.701581 21087 raft_consensus.cc:2454] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fbce56ead24df1b6eb307b25a9bdf4 in term 0.
I20260806 10:47:40.701867 20953 raft_consensus.cc:2454] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fbce56ead24df1b6eb307b25a9bdf4 in term 0.
I20260806 10:47:40.702246 20753 leader_election.cc:302] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cac1e5fe35a44098f1c31a1d03b3968, 36fbce56ead24df1b6eb307b25a9bdf4; no voters:
I20260806 10:47:40.702637 21159 raft_consensus.cc:2781] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:40.702790 21159 raft_consensus.cc:482] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:40.702921 21159 raft_consensus.cc:3037] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:40.705178 21159 raft_consensus.cc:504] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.705834 21159 leader_election.cc:288] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [CANDIDATE]: Term 1 election: Requested vote from peers 638c1c5325c94ad5ac6370c1d3665622 (127.16.21.2:45143), 1cac1e5fe35a44098f1c31a1d03b3968 (127.16.21.3:32857)
I20260806 10:47:40.706117 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c66206af88848ef991b8be6548487a6" candidate_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638c1c5325c94ad5ac6370c1d3665622"
I20260806 10:47:40.706331 20953 raft_consensus.cc:3037] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:40.706322 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c66206af88848ef991b8be6548487a6" candidate_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cac1e5fe35a44098f1c31a1d03b3968"
I20260806 10:47:40.706518 21087 raft_consensus.cc:3037] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:40.708587 20953 raft_consensus.cc:2454] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fbce56ead24df1b6eb307b25a9bdf4 in term 1.
I20260806 10:47:40.708834 21087 raft_consensus.cc:2454] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fbce56ead24df1b6eb307b25a9bdf4 in term 1.
I20260806 10:47:40.709051 20754 leader_election.cc:302] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fbce56ead24df1b6eb307b25a9bdf4, 638c1c5325c94ad5ac6370c1d3665622; no voters:
I20260806 10:47:40.709326 21159 raft_consensus.cc:2781] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:40.709721 21159 raft_consensus.cc:686] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 1 LEADER]: Becoming Leader. State: Replica: 36fbce56ead24df1b6eb307b25a9bdf4, State: Running, Role: LEADER
I20260806 10:47:40.710315 21159 consensus_queue.cc:237] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } }
I20260806 10:47:40.714749 20673 catalog_manager.cc:5495] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36fbce56ead24df1b6eb307b25a9bdf4 (127.16.21.1). New cstate: current_term: 1 leader_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fbce56ead24df1b6eb307b25a9bdf4" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 39005 } health_report { overall_health: HEALTHY } } }
I20260806 10:47:40.749927 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.744914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.16.21.1:39005)
0806 10:47:40.745085 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.745090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.745105 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:47:40.745206 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.745210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.745296 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.745405 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:40.745731 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.746454 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.747750 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.748183 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.748325 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.748761 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:40.749326 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:40.749477 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.749586 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.749663 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:47:40.757552 21087 raft_consensus.cc:1261] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 1 FOLLOWER]: Refusing update from remote peer 36fbce56ead24df1b6eb307b25a9bdf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:40.757604 20953 raft_consensus.cc:1261] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 1 FOLLOWER]: Refusing update from remote peer 36fbce56ead24df1b6eb307b25a9bdf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:40.758469 21164 consensus_queue.cc:1035] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cac1e5fe35a44098f1c31a1d03b3968" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:40.758809 21159 consensus_queue.cc:1035] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638c1c5325c94ad5ac6370c1d3665622" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:40.786228 20673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:40.786509 20673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:40.790232 20673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6aaedaf047744f6aa1f5da11e8950648: Sending DeleteTablet for 3 replicas of tablet 7c66206af88848ef991b8be6548487a6
I20260806 10:47:40.791231 21067 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c66206af88848ef991b8be6548487a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:40 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:55032
I20260806 10:47:40.791294 20932 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c66206af88848ef991b8be6548487a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:40 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:54138
I20260806 10:47:40.792043 21177 tablet_replica.cc:330] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: stopping tablet replica
I20260806 10:47:40.792143 20799 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c66206af88848ef991b8be6548487a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:40 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:42382
I20260806 10:47:40.792483 21177 raft_consensus.cc:2229] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:40.792955 21177 raft_consensus.cc:2258] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:40.793467 21178 tablet_replica.cc:330] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: stopping tablet replica
I20260806 10:47:40.793491 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20735
I20260806 10:47:40.793938 21178 raft_consensus.cc:2229] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:40.794023 21179 tablet_replica.cc:330] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: stopping tablet replica
I20260806 10:47:40.794373 21179 raft_consensus.cc:2229] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:40.794492 21178 raft_consensus.cc:2258] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:40.795401 21179 raft_consensus.cc:2258] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:40.796829 21178 ts_tablet_manager.cc:1905] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:40.798777 21177 ts_tablet_manager.cc:1905] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:40.804003 21178 ts_tablet_manager.cc:1918] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:40.804237 21178 log.cc:1196] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/7c66206af88848ef991b8be6548487a6
I20260806 10:47:40.804880 21178 ts_tablet_manager.cc:1939] T 7c66206af88848ef991b8be6548487a6 P 638c1c5325c94ad5ac6370c1d3665622: Deleting consensus metadata
I20260806 10:47:40.806749 20660 catalog_manager.cc:4841] TS 638c1c5325c94ad5ac6370c1d3665622 (127.16.21.2:45143): tablet 7c66206af88848ef991b8be6548487a6 (table test-table [id=d006b8ed57f54edaa02400cd62f67839]) successfully deleted
I20260806 10:47:40.807925 21177 ts_tablet_manager.cc:1918] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:40.808156 21177 log.cc:1196] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/7c66206af88848ef991b8be6548487a6
I20260806 10:47:40.808436 21179 ts_tablet_manager.cc:1905] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:40.808876 21177 ts_tablet_manager.cc:1939] T 7c66206af88848ef991b8be6548487a6 P 1cac1e5fe35a44098f1c31a1d03b3968: Deleting consensus metadata
I20260806 10:47:40.810949 20659 catalog_manager.cc:4841] TS 1cac1e5fe35a44098f1c31a1d03b3968 (127.16.21.3:32857): tablet 7c66206af88848ef991b8be6548487a6 (table test-table [id=d006b8ed57f54edaa02400cd62f67839]) successfully deleted
I20260806 10:47:40.816155 21179 ts_tablet_manager.cc:1918] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:40.816375 21179 log.cc:1196] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/7c66206af88848ef991b8be6548487a6
I20260806 10:47:40.817312 21179 ts_tablet_manager.cc:1939] T 7c66206af88848ef991b8be6548487a6 P 36fbce56ead24df1b6eb307b25a9bdf4: Deleting consensus metadata
I20260806 10:47:40.818972 20658 catalog_manager.cc:4841] TS 36fbce56ead24df1b6eb307b25a9bdf4 (127.16.21.1:39005): tablet 7c66206af88848ef991b8be6548487a6 (table test-table [id=d006b8ed57f54edaa02400cd62f67839]) successfully deleted
Aug 06 10:47:40 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:40.826674 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.795793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.16.21.1:39005)
0806 10:47:40.796005 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.796010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.811784 (+ 15774us) server_negotiation.cc:407] Connection header received
0806 10:47:40.812028 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.812034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.812120 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.812226 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:40.813578 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.814541 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.815747 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.816217 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.816402 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.820819 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:40.820830 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:40.820847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:40.820881 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:40.823785 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.824491 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.824498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.824500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.824658 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.825067 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.825072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.825077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.825425 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:40.825765 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.826058 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.826314 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:47:40.827355 20859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:40.926191 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20735
I20260806 10:47:40.948433 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20870
Aug 06 10:47:40 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:40.963248 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:40.949230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.16.21.2:45143)
0806 10:47:40.949434 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:40.949438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:40.949456 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:47:40.949620 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:40.949624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:40.949724 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:40.949816 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:40.951011 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.951780 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.953087 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:40.953493 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:40.953582 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:40.957561 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:40.957574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:40.957588 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:40.957618 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:40.960707 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.961266 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.961272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.961281 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.961411 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:40.961719 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:40.961722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:40.961724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:40.962096 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:40.962495 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:40.962654 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 10:47:40.962905 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 10:47:40.963935 20993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:41.053050 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20870
I20260806 10:47:41.073711 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21004
Aug 06 10:47:41 dist-test-slave-0b58 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:41.088889 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:41.074530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.16.21.3:32857)
0806 10:47:41.074718 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:41.074726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:41.074750 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:47:41.075007 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:41.075011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:41.075093 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:41.075211 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:41.076588 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:41.077304 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:41.078507 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:41.078889 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:41.079027 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:41.083639 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:41.083656 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:41.083672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:41.083706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:41.086644 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:41.087195 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:41.087202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:41.087205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:41.087329 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:41.087639 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:41.087642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:41.087644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:41.087987 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:41.088208 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:41.088469 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 10:47:41.088629 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:47:41.089538 21127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:41.177443 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21004
I20260806 10:47:41.197551 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 20642
I20260806 10:47:41.198756 20703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:41.311268 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 20642
2026-08-06T10:47:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4377 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T10:47:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:41Z Disabled control of system clock
I20260806 10:47:41.390100 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:43443
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 10:47:41.711431 21188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:41.711691 21188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:41.711776 21188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:41.718845 21188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:41.718942 21188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:41.718981 21188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:41.719051 21188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:41.719095 21188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:41.727882 21188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:43443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:41.729095 21188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:41.730751 21188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:41.742100 21194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:41.742213 21193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:41.742208 21196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:41.743371 21188 server_base.cc:1034] running on GCE node
I20260806 10:47:41.744168 21188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:41.745541 21188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:41.746779 21188 hybrid_clock.cc:648] HybridClock initialized: now 1786013261746731 us; error 34 us; skew 500 ppm
I20260806 10:47:41.747112 21188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:41.749330 21188 webserver.cc:460] Webserver started at http://127.16.21.62:37309/ using document root <none> and password file <none>
I20260806 10:47:41.749991 21188 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:41.750126 21188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:41.750453 21188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:41.753582 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "dd25e718c07b43c59da664f11bbc8f42"
format_stamp: "Formatted at 2026-08-06 10:47:41 on dist-test-slave-0b58"
server_key: "f95dc3adf995c63fc5b4b058d8d8dcfa"
server_key_iv: "dcbaff21ef54fe06a071aeeae9a12d0d"
server_key_version: "encryptionkey@0"
I20260806 10:47:41.754575 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "dd25e718c07b43c59da664f11bbc8f42"
format_stamp: "Formatted at 2026-08-06 10:47:41 on dist-test-slave-0b58"
server_key: "f95dc3adf995c63fc5b4b058d8d8dcfa"
server_key_iv: "dcbaff21ef54fe06a071aeeae9a12d0d"
server_key_version: "encryptionkey@0"
I20260806 10:47:41.759377 21188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:47:41.762585 21202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:41.763827 21188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:47:41.764008 21188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "dd25e718c07b43c59da664f11bbc8f42"
format_stamp: "Formatted at 2026-08-06 10:47:41 on dist-test-slave-0b58"
server_key: "f95dc3adf995c63fc5b4b058d8d8dcfa"
server_key_iv: "dcbaff21ef54fe06a071aeeae9a12d0d"
server_key_version: "encryptionkey@0"
I20260806 10:47:41.764312 21188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:41.769802 21188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:41.770538 21188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:41.770839 21188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:41.789702 21188 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:43443
I20260806 10:47:41.789783 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:43443 every 8 connection(s)
I20260806 10:47:41.790969 21188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:41.794509 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:41.797269 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21188
I20260806 10:47:41.797466 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:41.798343 16468 external_mini_cluster.cc:1428] Setting key d377e987d3bfec15ef9e9a72f2f2f6d0
I20260806 10:47:41.806674 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:41.800489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.16.21.62:43443)
0806 10:47:41.801461 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:41.801470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:41.801509 (+ 39us) server_negotiation.cc:407] Connection header received
0806 10:47:41.803151 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:41.803171 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:41.803639 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:41.804090 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:41.805171 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:41.805181 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:41.805192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:41.805205 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:41.805264 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:41.805283 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:41.805458 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:41.805568 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 10:47:41.805832 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":377,"threads_started":1}
I20260806 10:47:41.807595 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42: Bootstrap starting.
I20260806 10:47:41.810900 21254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:41.811862 21254 log.cc:824] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:41.814512 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42: No bootstrap required, opened a new log
I20260806 10:47:41.821502 21254 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd25e718c07b43c59da664f11bbc8f42" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43443 } }
I20260806 10:47:41.821853 21254 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:41.821969 21254 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd25e718c07b43c59da664f11bbc8f42, State: Initialized, Role: FOLLOWER
I20260806 10:47:41.822474 21254 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd25e718c07b43c59da664f11bbc8f42" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43443 } }
I20260806 10:47:41.822710 21254 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:41.822854 21254 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:41.823015 21254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:41.825143 21254 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd25e718c07b43c59da664f11bbc8f42" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43443 } }
I20260806 10:47:41.825599 21254 leader_election.cc:302] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd25e718c07b43c59da664f11bbc8f42; no voters:
I20260806 10:47:41.826184 21254 leader_election.cc:288] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:41.826289 21259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:41.827123 21259 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [term 1 LEADER]: Becoming Leader. State: Replica: dd25e718c07b43c59da664f11bbc8f42, State: Running, Role: LEADER
I20260806 10:47:41.827637 21259 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd25e718c07b43c59da664f11bbc8f42" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43443 } }
I20260806 10:47:41.828184 21254 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:41.830622 21260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd25e718c07b43c59da664f11bbc8f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd25e718c07b43c59da664f11bbc8f42" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43443 } } }
I20260806 10:47:41.831058 21260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:41.830902 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd25e718c07b43c59da664f11bbc8f42. Latest consensus state: current_term: 1 leader_uuid: "dd25e718c07b43c59da664f11bbc8f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd25e718c07b43c59da664f11bbc8f42" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43443 } } }
I20260806 10:47:41.831249 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:41.832901 21268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:41.837522 21268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:41.845489 21268 catalog_manager.cc:1347] Generated new cluster ID: 270b066c6f774261b8c5b1779585d462
I20260806 10:47:41.845600 21268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:41.868695 21268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:41.869133 21268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:41.878993 21268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42: Generated new TSK 0
I20260806 10:47:41.879899 21268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:47:41.890748 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43443
--builtin_ntp_servers=127.16.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 10:47:42.217846 21279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:42.218214 21279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:42.218379 21279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:42.229753 21279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:42.229864 21279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:42.230064 21279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:42.239920 21279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:42.241226 21279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:42.242974 21279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:42.255489 21284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:42.255431 21285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:42.255529 21287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:42.256829 21279 server_base.cc:1034] running on GCE node
I20260806 10:47:42.257369 21279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:42.258220 21279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:42.259438 21279 hybrid_clock.cc:648] HybridClock initialized: now 1786013262259393 us; error 54 us; skew 500 ppm
I20260806 10:47:42.259729 21279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:42.261880 21279 webserver.cc:460] Webserver started at http://127.16.21.1:41179/ using document root <none> and password file <none>
I20260806 10:47:42.262552 21279 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:42.262636 21279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:42.262909 21279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:42.265659 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "3c084a114aa042b9a0b15030f450b5cb"
format_stamp: "Formatted at 2026-08-06 10:47:42 on dist-test-slave-0b58"
server_key: "ccc210db8b87a73489e5b4221d9ae4da"
server_key_iv: "fd71092b1d540daa4288fbb0496b5cba"
server_key_version: "encryptionkey@0"
I20260806 10:47:42.266487 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "3c084a114aa042b9a0b15030f450b5cb"
format_stamp: "Formatted at 2026-08-06 10:47:42 on dist-test-slave-0b58"
server_key: "ccc210db8b87a73489e5b4221d9ae4da"
server_key_iv: "fd71092b1d540daa4288fbb0496b5cba"
server_key_version: "encryptionkey@0"
I20260806 10:47:42.271400 21279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:47:42.274672 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:42.275933 21279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:42.276072 21279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "3c084a114aa042b9a0b15030f450b5cb"
format_stamp: "Formatted at 2026-08-06 10:47:42 on dist-test-slave-0b58"
server_key: "ccc210db8b87a73489e5b4221d9ae4da"
server_key_iv: "fd71092b1d540daa4288fbb0496b5cba"
server_key_version: "encryptionkey@0"
I20260806 10:47:42.276244 21279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:42.281795 21279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:42.282635 21279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:42.282966 21279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:42.284669 21279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:42.286052 21279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:42.286147 21279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:42.286242 21279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:42.286298 21279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:42.319031 21279 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:42455
I20260806 10:47:42.319118 21407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:42455 every 8 connection(s)
I20260806 10:47:42.320372 21279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:42.323840 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21279
I20260806 10:47:42.324038 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:42.324865 16468 external_mini_cluster.cc:1428] Setting key e6e83af1a1ad8d1ea3cf9e0837b0cef0
I20260806 10:47:42.327880 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43443
--builtin_ntp_servers=127.16.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 10:47:42.329288 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:42.322882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:39167 (local address 127.16.21.62:43443)
0806 10:47:42.323320 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:42.323324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:42.324707 (+ 1383us) server_negotiation.cc:407] Connection header received
0806 10:47:42.326777 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:42.326780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:42.326826 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:42.326950 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:42.328363 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:42.328391 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:42.328398 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:42.328420 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:42.328492 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:42.328510 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:42.328633 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:42.328818 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 10:47:42.328885 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":227,"threads_started":1}
I20260806 10:47:42.330089 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:42.323447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:43443 (local address 127.16.21.1:39167)
0806 10:47:42.324540 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:42.324583 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:42.326470 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:42.327155 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:42.327162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:42.327244 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:42.327988 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:42.327991 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:42.328051 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:42.328161 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:42.328620 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:42.328628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:42.328633 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:47:42.328844 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 10:47:42.329069 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":339,"threads_started":1}
I20260806 10:47:42.331513 21408 heartbeater.cc:347] Connected to a master server at 127.16.21.62:43443
I20260806 10:47:42.331802 21408 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:42.332530 21408 heartbeater.cc:511] Master 127.16.21.62:43443 requested a full tablet report, sending...
I20260806 10:47:42.334481 21219 ts_manager.cc:194] Registered new tserver with Master: 3c084a114aa042b9a0b15030f450b5cb (127.16.21.1:42455)
W20260806 10:47:42.652588 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:42.652848 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:42.653013 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:42.660033 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:42.660120 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:42.660265 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:42.669494 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:42.670715 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:42.672506 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:42.685681 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:42.685741 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:42.685763 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:42.686544 21412 server_base.cc:1034] running on GCE node
I20260806 10:47:42.687110 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:42.687978 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:42.689188 21412 hybrid_clock.cc:648] HybridClock initialized: now 1786013262689144 us; error 35 us; skew 500 ppm
I20260806 10:47:42.689513 21412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:42.691685 21412 webserver.cc:460] Webserver started at http://127.16.21.2:41901/ using document root <none> and password file <none>
I20260806 10:47:42.692392 21412 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:42.692505 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:42.692855 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:42.695604 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "abdf21c2a4e447a6a27c1459ddf0e40c"
format_stamp: "Formatted at 2026-08-06 10:47:42 on dist-test-slave-0b58"
server_key: "17561bc8cfaba46fe7a71d97dec0460e"
server_key_iv: "25dbe89b26e97c6061b08acd995b92a1"
server_key_version: "encryptionkey@0"
I20260806 10:47:42.696452 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "abdf21c2a4e447a6a27c1459ddf0e40c"
format_stamp: "Formatted at 2026-08-06 10:47:42 on dist-test-slave-0b58"
server_key: "17561bc8cfaba46fe7a71d97dec0460e"
server_key_iv: "25dbe89b26e97c6061b08acd995b92a1"
server_key_version: "encryptionkey@0"
I20260806 10:47:42.701313 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:47:42.704640 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:42.705828 21412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:47:42.705993 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "abdf21c2a4e447a6a27c1459ddf0e40c"
format_stamp: "Formatted at 2026-08-06 10:47:42 on dist-test-slave-0b58"
server_key: "17561bc8cfaba46fe7a71d97dec0460e"
server_key_iv: "25dbe89b26e97c6061b08acd995b92a1"
server_key_version: "encryptionkey@0"
I20260806 10:47:42.706218 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:42.711733 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:42.712530 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:42.712805 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:42.713665 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:42.714938 21412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:42.715057 21412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:42.715217 21412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:42.715312 21412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:42.748175 21412 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:41229
I20260806 10:47:42.748297 21538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:41229 every 8 connection(s)
I20260806 10:47:42.749471 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:42.752226 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21412
I20260806 10:47:42.752445 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:42.753033 16468 external_mini_cluster.cc:1428] Setting key 3d7c31e2e5818e45cd8d37bdf4ea6c24
I20260806 10:47:42.756182 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43443
--builtin_ntp_servers=127.16.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 10:47:42.758085 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:42.751891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:35689 (local address 127.16.21.62:43443)
0806 10:47:42.752117 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:42.752123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:42.753746 (+ 1623us) server_negotiation.cc:407] Connection header received
0806 10:47:42.755799 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:42.755802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:42.755862 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:42.755941 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:42.757429 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:42.757433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:42.757437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:42.757451 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:42.757482 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:42.757494 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:42.757584 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:42.757819 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 10:47:42.757869 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:47:42.759284 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:42.752482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:43443 (local address 127.16.21.2:35689)
0806 10:47:42.753569 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:42.753617 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:42.755514 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:42.756205 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:42.756216 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:42.756368 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:42.757075 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:42.757081 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:42.757131 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:42.757244 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:42.757595 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:42.757610 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:42.757620 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:47:42.757899 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 10:47:42.758135 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":290,"threads_started":1}
I20260806 10:47:42.760619 21539 heartbeater.cc:347] Connected to a master server at 127.16.21.62:43443
I20260806 10:47:42.760910 21539 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:42.761438 21539 heartbeater.cc:511] Master 127.16.21.62:43443 requested a full tablet report, sending...
I20260806 10:47:42.762809 21219 ts_manager.cc:194] Registered new tserver with Master: abdf21c2a4e447a6a27c1459ddf0e40c (127.16.21.2:41229)
W20260806 10:47:43.080090 21543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:43.080367 21543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:43.080538 21543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:43.087431 21543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:43.087513 21543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:43.087700 21543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:43.096283 21543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:43.097395 21543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:43.099092 21543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:43.111817 21550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:43.111896 21549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:43.111845 21552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:43.112900 21543 server_base.cc:1034] running on GCE node
I20260806 10:47:43.113487 21543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:43.114339 21543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:43.115557 21543 hybrid_clock.cc:648] HybridClock initialized: now 1786013263115530 us; error 43 us; skew 500 ppm
I20260806 10:47:43.115873 21543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:43.118309 21543 webserver.cc:460] Webserver started at http://127.16.21.3:34819/ using document root <none> and password file <none>
I20260806 10:47:43.118996 21543 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:43.119133 21543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:43.119460 21543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:43.122238 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "06113aa39c5a4efa82ceb4f38e9794a5"
format_stamp: "Formatted at 2026-08-06 10:47:43 on dist-test-slave-0b58"
server_key: "64d22e5f51cfa27b8ec461fe05915c92"
server_key_iv: "08f2a1d34173ca7a3652451da7f8a7ba"
server_key_version: "encryptionkey@0"
I20260806 10:47:43.123024 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "06113aa39c5a4efa82ceb4f38e9794a5"
format_stamp: "Formatted at 2026-08-06 10:47:43 on dist-test-slave-0b58"
server_key: "64d22e5f51cfa27b8ec461fe05915c92"
server_key_iv: "08f2a1d34173ca7a3652451da7f8a7ba"
server_key_version: "encryptionkey@0"
I20260806 10:47:43.127898 21543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 10:47:43.131304 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:43.132579 21543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:47:43.132761 21543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "06113aa39c5a4efa82ceb4f38e9794a5"
format_stamp: "Formatted at 2026-08-06 10:47:43 on dist-test-slave-0b58"
server_key: "64d22e5f51cfa27b8ec461fe05915c92"
server_key_iv: "08f2a1d34173ca7a3652451da7f8a7ba"
server_key_version: "encryptionkey@0"
I20260806 10:47:43.132977 21543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:43.138378 21543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:43.139115 21543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:43.139370 21543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:43.140681 21543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:43.141980 21543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:43.142100 21543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:43.142231 21543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:43.142328 21543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:43.175330 21543 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:40913
I20260806 10:47:43.175416 21676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:40913 every 8 connection(s)
I20260806 10:47:43.176659 21543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:43.180482 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21543
I20260806 10:47:43.180647 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:43.181262 16468 external_mini_cluster.cc:1428] Setting key 4ef804757be58851a4ee4bd42fbb76b8
I20260806 10:47:43.185628 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.179122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:46735 (local address 127.16.21.62:43443)
0806 10:47:43.179334 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.179338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.180905 (+ 1567us) server_negotiation.cc:407] Connection header received
0806 10:47:43.183377 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.183385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.183456 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.183545 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.185000 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.185004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.185008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.185023 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.185055 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.185067 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.185164 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.185374 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.185424 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:47:43.186515 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.179572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:43443 (local address 127.16.21.3:46735)
0806 10:47:43.180730 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.180787 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:43.183054 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:43.183801 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:43.183809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:43.183896 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:43.184675 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:43.184680 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:43.184732 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:43.184847 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:43.185182 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:43.185191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:43.185196 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:47:43.185423 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 10:47:43.185599 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":266,"threads_started":1}
I20260806 10:47:43.187711 21677 heartbeater.cc:347] Connected to a master server at 127.16.21.62:43443
I20260806 10:47:43.187996 21677 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:43.188612 21677 heartbeater.cc:511] Master 127.16.21.62:43443 requested a full tablet report, sending...
I20260806 10:47:43.189908 21219 ts_manager.cc:194] Registered new tserver with Master: 06113aa39c5a4efa82ceb4f38e9794a5 (127.16.21.3:40913)
I20260806 10:47:43.196638 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:47:43.203152 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.200929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.16.21.62:43443)
0806 10:47:43.201080 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.201084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.201626 (+ 542us) server_negotiation.cc:407] Connection header received
0806 10:47:43.201743 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.201746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.201800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.201875 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.202624 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.202628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.202631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.202645 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.202675 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.202686 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.202784 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.202815 (+ 31us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.202856 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:47:43.206712 21219 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:43.213688 21219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:43.245571 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.225031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:41229 (local address 127.0.0.1:51928)
0806 10:47:43.241690 (+ 16659us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.241715 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:43.241872 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:43.242553 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:43.242562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:43.242583 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:43.244030 (+ 1447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:43.244036 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:43.244053 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:43.244095 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:43.244598 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:43.244610 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:43.244619 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:47:43.244820 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 10:47:43.244923 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16471,"thread_start_us":17302,"threads_started":1}
I20260806 10:47:43.245577 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.224919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60596)
0806 10:47:43.240209 (+ 15290us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.241002 (+ 793us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:43.241180 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:43.242401 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:43.242415 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:43.242510 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:43.243538 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:43.243542 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:43.243574 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:43.243664 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:43.244781 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:43.244788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:43.244794 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 10:47:43.244920 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 10:47:43.244958 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14967,"thread_start_us":15354,"threads_started":1}
I20260806 10:47:43.246003 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.225723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.16.21.2:41229)
0806 10:47:43.234368 (+ 8645us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.234380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.241852 (+ 7472us) server_negotiation.cc:407] Connection header received
0806 10:47:43.242009 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.242015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.242176 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.242314 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.244331 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.244349 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.244364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.244391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.244454 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.244479 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.244659 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.245320 (+ 661us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.245427 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8414,"thread_start_us":8248,"threads_started":1}
I20260806 10:47:43.246184 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.225463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:40913 (local address 127.0.0.1:56992)
0806 10:47:43.242574 (+ 17111us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.242590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:43.242824 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:43.243576 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:43.243580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:43.243601 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:43.244786 (+ 1185us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:43.244789 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:43.244807 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:43.244847 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:43.245407 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:43.245411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:43.245424 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 10:47:43.245589 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 10:47:43.245642 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17001,"thread_start_us":16518,"threads_started":1}
I20260806 10:47:43.246598 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.225020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.16.21.1:42455)
0806 10:47:43.241760 (+ 16740us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.241766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.241819 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:47:43.241897 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.241905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.242153 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.243089 (+ 936us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.244513 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.244519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.244528 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.244578 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.244632 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.244653 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.246037 (+ 1384us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.246125 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.246235 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16584,"thread_start_us":16463,"threads_started":1}
I20260806 10:47:43.246876 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.226057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.16.21.3:40913)
0806 10:47:43.235512 (+ 9455us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.235523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.242719 (+ 7196us) server_negotiation.cc:407] Connection header received
0806 10:47:43.243019 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.243029 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.243231 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.243346 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.245029 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.245038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.245050 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.245072 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.245134 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.245157 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.245340 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.246420 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.246526 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9252,"thread_start_us":9072,"threads_started":1}
I20260806 10:47:43.249567 21474 tablet_service.cc:1467] Processing CreateTablet for tablet c87c124f8db34b72b50065e604fc5fd4 (DEFAULT_TABLE table=test-table [id=45cbf651468c4291afa6aeeee485c74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:43.250092 21609 tablet_service.cc:1467] Processing CreateTablet for tablet c87c124f8db34b72b50065e604fc5fd4 (DEFAULT_TABLE table=test-table [id=45cbf651468c4291afa6aeeee485c74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:43.250084 21341 tablet_service.cc:1467] Processing CreateTablet for tablet c87c124f8db34b72b50065e604fc5fd4 (DEFAULT_TABLE table=test-table [id=45cbf651468c4291afa6aeeee485c74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:43.250969 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87c124f8db34b72b50065e604fc5fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:43.251375 21341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87c124f8db34b72b50065e604fc5fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:43.251397 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87c124f8db34b72b50065e604fc5fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:43.261938 21695 tablet_bootstrap.cc:492] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Bootstrap starting.
I20260806 10:47:43.264873 21695 tablet_bootstrap.cc:654] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:43.265882 21696 tablet_bootstrap.cc:492] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Bootstrap starting.
I20260806 10:47:43.266884 21697 tablet_bootstrap.cc:492] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: Bootstrap starting.
I20260806 10:47:43.267720 21695 log.cc:824] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:43.269670 21696 tablet_bootstrap.cc:654] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:43.270129 21697 tablet_bootstrap.cc:654] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:43.270876 21696 log.cc:824] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:43.270948 21695 tablet_bootstrap.cc:492] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: No bootstrap required, opened a new log
I20260806 10:47:43.271104 21697 log.cc:824] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:43.271190 21695 ts_tablet_manager.cc:1397] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260806 10:47:43.273532 21696 tablet_bootstrap.cc:492] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: No bootstrap required, opened a new log
I20260806 10:47:43.273787 21696 ts_tablet_manager.cc:1397] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:47:43.274010 21697 tablet_bootstrap.cc:492] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: No bootstrap required, opened a new log
I20260806 10:47:43.274246 21697 ts_tablet_manager.cc:1397] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260806 10:47:43.281569 21696 raft_consensus.cc:348] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.281909 21696 raft_consensus.cc:374] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:43.282033 21696 raft_consensus.cc:729] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06113aa39c5a4efa82ceb4f38e9794a5, State: Initialized, Role: FOLLOWER
I20260806 10:47:43.282238 21695 raft_consensus.cc:348] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.282644 21695 raft_consensus.cc:374] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:43.282738 21695 raft_consensus.cc:729] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdf21c2a4e447a6a27c1459ddf0e40c, State: Initialized, Role: FOLLOWER
I20260806 10:47:43.282630 21696 consensus_queue.cc:260] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.283366 21695 consensus_queue.cc:260] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.283753 21677 heartbeater.cc:503] Master 127.16.21.62:43443 was elected leader, sending a full tablet report...
I20260806 10:47:43.284469 21539 heartbeater.cc:503] Master 127.16.21.62:43443 was elected leader, sending a full tablet report...
I20260806 10:47:43.284582 21697 raft_consensus.cc:348] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.285019 21697 raft_consensus.cc:374] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:43.285110 21697 raft_consensus.cc:729] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c084a114aa042b9a0b15030f450b5cb, State: Initialized, Role: FOLLOWER
I20260806 10:47:43.285709 21697 consensus_queue.cc:260] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.288956 21696 ts_tablet_manager.cc:1428] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
I20260806 10:47:43.289711 21695 ts_tablet_manager.cc:1428] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260806 10:47:43.290040 21408 heartbeater.cc:503] Master 127.16.21.62:43443 was elected leader, sending a full tablet report...
I20260806 10:47:43.290674 21697 ts_tablet_manager.cc:1428] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260806 10:47:43.304049 21701 raft_consensus.cc:482] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:43.304239 21701 raft_consensus.cc:504] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.305871 21701 leader_election.cc:288] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c084a114aa042b9a0b15030f450b5cb (127.16.21.1:42455), abdf21c2a4e447a6a27c1459ddf0e40c (127.16.21.2:41229)
I20260806 10:47:43.308578 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.306465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:57815 (local address 127.16.21.2:41229)
0806 10:47:43.306670 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.306678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.306702 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:47:43.306761 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.306766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.306820 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.306917 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.307962 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.307967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.307974 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.307995 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.308023 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.308034 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.308158 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.308249 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.308337 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:47:43.308579 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.306107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:41229 (local address 127.16.21.3:57815)
0806 10:47:43.306356 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.306376 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:43.306472 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:43.307036 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:43.307041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:43.307095 (+ 54us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:43.307768 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:43.307770 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:43.307788 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:43.307827 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:43.308127 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:43.308131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:43.308137 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 10:47:43.308290 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 10:47:43.308340 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 10:47:43.309147 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.306275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:49327 (local address 127.16.21.1:42455)
0806 10:47:43.306434 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.306439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.307102 (+ 663us) server_negotiation.cc:407] Connection header received
0806 10:47:43.307240 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.307246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.307314 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.307410 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.308486 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.308491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.308496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.308517 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.308552 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.308563 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.308680 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.308835 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.308892 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:47:43.309336 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.306180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.16.21.3:49327)
0806 10:47:43.306995 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.307012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:43.307116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:43.307515 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:43.307522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:43.307549 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:43.308230 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:47:43.308233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:43.308245 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:47:43.308348 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:43.308658 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:43.308663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:43.308667 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 10:47:43.308835 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 10:47:43.308890 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":525,"threads_started":1}
I20260806 10:47:43.309788 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87c124f8db34b72b50065e604fc5fd4" candidate_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" is_pre_election: true
I20260806 10:47:43.310330 21494 raft_consensus.cc:2454] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06113aa39c5a4efa82ceb4f38e9794a5 in term 0.
I20260806 10:47:43.310251 21362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87c124f8db34b72b50065e604fc5fd4" candidate_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c084a114aa042b9a0b15030f450b5cb" is_pre_election: true
I20260806 10:47:43.310779 21362 raft_consensus.cc:2454] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06113aa39c5a4efa82ceb4f38e9794a5 in term 0.
I20260806 10:47:43.311007 21564 leader_election.cc:302] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06113aa39c5a4efa82ceb4f38e9794a5, abdf21c2a4e447a6a27c1459ddf0e40c; no voters:
I20260806 10:47:43.311368 21701 raft_consensus.cc:2781] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:43.311537 21701 raft_consensus.cc:482] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:43.311672 21701 raft_consensus.cc:3037] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:43.313856 21701 raft_consensus.cc:504] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
I20260806 10:47:43.314395 21701 leader_election.cc:288] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [CANDIDATE]: Term 1 election: Requested vote from peers 3c084a114aa042b9a0b15030f450b5cb (127.16.21.1:42455), abdf21c2a4e447a6a27c1459ddf0e40c (127.16.21.2:41229)
I20260806 10:47:43.314808 21362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87c124f8db34b72b50065e604fc5fd4" candidate_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c084a114aa042b9a0b15030f450b5cb"
I20260806 10:47:43.314831 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87c124f8db34b72b50065e604fc5fd4" candidate_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c"
I20260806 10:47:43.314987 21362 raft_consensus.cc:3037] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:43.315002 21494 raft_consensus.cc:3037] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:43.317198 21362 raft_consensus.cc:2454] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06113aa39c5a4efa82ceb4f38e9794a5 in term 1.
I20260806 10:47:43.317252 21494 raft_consensus.cc:2454] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06113aa39c5a4efa82ceb4f38e9794a5 in term 1.
I20260806 10:47:43.317662 21564 leader_election.cc:302] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06113aa39c5a4efa82ceb4f38e9794a5, abdf21c2a4e447a6a27c1459ddf0e40c; no voters:
I20260806 10:47:43.317945 21701 raft_consensus.cc:2781] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:43.318394 21701 raft_consensus.cc:686] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 1 LEADER]: Becoming Leader. State: Replica: 06113aa39c5a4efa82ceb4f38e9794a5, State: Running, Role: LEADER
I20260806 10:47:43.318892 21701 consensus_queue.cc:237] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } }
W20260806 10:47:43.322654 21409 tablet.cc:2369] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:43.323618 21218 catalog_manager.cc:5495] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06113aa39c5a4efa82ceb4f38e9794a5 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06113aa39c5a4efa82ceb4f38e9794a5" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40913 } health_report { overall_health: HEALTHY } } }
I20260806 10:47:43.359489 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.357278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.16.21.3:40913)
0806 10:47:43.357458 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.357464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.357575 (+ 111us) server_negotiation.cc:407] Connection header received
0806 10:47:43.357810 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.357814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.357887 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.357997 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.358892 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.358898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.358902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.358920 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.358961 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.358973 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.359098 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.359187 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.359245 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:47:43.367303 21362 raft_consensus.cc:1261] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb [term 1 FOLLOWER]: Refusing update from remote peer 06113aa39c5a4efa82ceb4f38e9794a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:43.367290 21494 raft_consensus.cc:1261] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 1 FOLLOWER]: Refusing update from remote peer 06113aa39c5a4efa82ceb4f38e9794a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:43.368296 21705 consensus_queue.cc:1035] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c084a114aa042b9a0b15030f450b5cb" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:43.368678 21701 consensus_queue.cc:1035] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abdf21c2a4e447a6a27c1459ddf0e40c" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:43.381340 21710 mvcc.cc:205] Tried to move back new op lower bound from 7315510326742474752 to 7315510326558113792. Current Snapshot: MvccSnapshot[applied={T|T < 7315510326742474752}]
I20260806 10:47:43.385159 21709 mvcc.cc:205] Tried to move back new op lower bound from 7315510326742474752 to 7315510326558113792. Current Snapshot: MvccSnapshot[applied={T|T < 7315510326742474752}]
I20260806 10:47:43.390966 21218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:43.391229 21218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:43.394601 21218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd25e718c07b43c59da664f11bbc8f42: Sending DeleteTablet for 3 replicas of tablet c87c124f8db34b72b50065e604fc5fd4
I20260806 10:47:43.395617 21341 tablet_service.cc:1514] Processing DeleteTablet for tablet c87c124f8db34b72b50065e604fc5fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:43 UTC) from {username='slave'} at 127.0.0.1:60596
I20260806 10:47:43.395618 21609 tablet_service.cc:1514] Processing DeleteTablet for tablet c87c124f8db34b72b50065e604fc5fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:43 UTC) from {username='slave'} at 127.0.0.1:56992
I20260806 10:47:43.395632 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet c87c124f8db34b72b50065e604fc5fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:43 UTC) from {username='slave'} at 127.0.0.1:51928
I20260806 10:47:43.401026 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21279
I20260806 10:47:43.409904 21719 tablet_replica.cc:330] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: stopping tablet replica
I20260806 10:47:43.410380 21719 raft_consensus.cc:2229] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:43.411890 21719 raft_consensus.cc:2258] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:43.414858 21719 ts_tablet_manager.cc:1905] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:43.422238 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.402857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.16.21.1:42455)
0806 10:47:43.403085 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.403093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.419488 (+ 16395us) server_negotiation.cc:407] Connection header received
0806 10:47:43.419969 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.419977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.420108 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.420179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.421449 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.421457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.421462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.421483 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.421520 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.421536 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.421628 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.421871 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.421940 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:47:43.423163 21719 ts_tablet_manager.cc:1918] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:43.423241 21403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:43.423337 21719 log.cc:1196] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/c87c124f8db34b72b50065e604fc5fd4
I20260806 10:47:43.424007 21719 ts_tablet_manager.cc:1939] T c87c124f8db34b72b50065e604fc5fd4 P abdf21c2a4e447a6a27c1459ddf0e40c: Deleting consensus metadata
I20260806 10:47:43.426254 21206 catalog_manager.cc:4841] TS abdf21c2a4e447a6a27c1459ddf0e40c (127.16.21.2:41229): tablet c87c124f8db34b72b50065e604fc5fd4 (table test-table [id=45cbf651468c4291afa6aeeee485c74e]) successfully deleted
I20260806 10:47:43.429078 21717 tablet_replica.cc:330] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: stopping tablet replica
I20260806 10:47:43.429726 21717 raft_consensus.cc:2229] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:43.430516 21717 raft_consensus.cc:2258] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:43.433156 21717 ts_tablet_manager.cc:1905] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:43.439002 21717 ts_tablet_manager.cc:1918] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:43.439177 21717 log.cc:1196] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/c87c124f8db34b72b50065e604fc5fd4
I20260806 10:47:43.439659 21717 ts_tablet_manager.cc:1939] T c87c124f8db34b72b50065e604fc5fd4 P 06113aa39c5a4efa82ceb4f38e9794a5: Deleting consensus metadata
I20260806 10:47:43.441107 21205 catalog_manager.cc:4841] TS 06113aa39c5a4efa82ceb4f38e9794a5 (127.16.21.3:40913): tablet c87c124f8db34b72b50065e604fc5fd4 (table test-table [id=45cbf651468c4291afa6aeeee485c74e]) successfully deleted
I20260806 10:47:43.582412 21718 tablet_replica.cc:330] T c87c124f8db34b72b50065e604fc5fd4 P 3c084a114aa042b9a0b15030f450b5cb: stopping tablet replica
I20260806 10:47:43.582643 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21279
W20260806 10:47:43.602873 21204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.21.1:42455 (error 108)
I20260806 10:47:43.603168 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21412
I20260806 10:47:43.605041 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.604449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60620)
0806 10:47:43.604716 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.604870 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260806 10:47:43.605355 21204 catalog_manager.cc:4584] TS 3c084a114aa042b9a0b15030f450b5cb (127.16.21.1:42455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c87c124f8db34b72b50065e604fc5fd4: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
I20260806 10:47:43.606583 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.603906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.16.21.2:41229)
0806 10:47:43.604133 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.604141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.604451 (+ 310us) server_negotiation.cc:407] Connection header received
0806 10:47:43.604633 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.604638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.604754 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.604876 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.605943 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.605949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.605957 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.605976 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.606018 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.606035 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.606188 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.606236 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.606296 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:47:43.607324 21534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:43.659073 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.658499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60632)
0806 10:47:43.658753 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.658886 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 10:47:43.660328 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.659854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60634)
0806 10:47:43.660036 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.660133 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 10:47:43.697693 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21412
I20260806 10:47:43.721130 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.720674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60646)
0806 10:47:43.720877 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.720970 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:47:43.722214 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.721858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60652)
0806 10:47:43.722014 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.722092 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:47:43.724805 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21543
I20260806 10:47:43.727946 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.725548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.16.21.3:40913)
0806 10:47:43.725797 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:43.725804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:43.725824 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:47:43.725934 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:43.725938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:43.725991 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:43.726093 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:43.727193 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:43.727199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:43.727204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:43.727225 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:43.727288 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:43.727307 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:43.727439 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:43.727578 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 10:47:43.727646 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 10:47:43.728790 21671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:43.803049 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.802499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60654)
0806 10:47:43.802733 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.802876 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 10:47:43.804380 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.803820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60656)
0806 10:47:43.804107 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.804196 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260806 10:47:43.849596 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21543
I20260806 10:47:43.872823 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21188
I20260806 10:47:43.873998 21249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:43.982992 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:43.982331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42455 (local address 127.0.0.1:60666)
0806 10:47:43.982613 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:43.982737 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 10:47:43.983140 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21188
2026-08-06T10:47:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2671 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 10:47:44.029707 16468 mini_oidc.cc:152] Starting JWKS server
I20260806 10:47:44.032639 16468 webserver.cc:460] Webserver started at https://127.0.0.1:34055/ using document root <none> and password file <none>
I20260806 10:47:44.033002 16468 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 10:47:44.033627 16468 webserver.cc:460] Webserver started at http://127.0.0.1:46591/ using document root <none> and password file <none>
2026-08-06T10:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:44Z Disabled control of system clock
I20260806 10:47:44.069821 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35409
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34055/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/testchainca.cert with env {}
W20260806 10:47:44.400079 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:44.400449 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:44.400620 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:44.407829 21731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:44.407968 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:44.408061 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:44.408155 21731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:44.408247 21731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:47:44.408381 21731 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:47:44.408444 21731 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34055/jwks.json
I20260806 10:47:44.418156 21731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34055/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:44.419413 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:44.420717 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:44.433068 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:44.433143 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:44.433143 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:44.433853 21731 server_base.cc:1034] running on GCE node
I20260806 10:47:44.434617 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:44.436007 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:44.437235 21731 hybrid_clock.cc:648] HybridClock initialized: now 1786013264437149 us; error 72 us; skew 500 ppm
I20260806 10:47:44.437587 21731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:44.439888 21731 webserver.cc:460] Webserver started at http://127.16.21.62:46697/ using document root <none> and password file <none>
I20260806 10:47:44.440620 21731 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:44.440757 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:44.441071 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:44.443876 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "286aa4818a234c6c9c160433d46d8c58"
format_stamp: "Formatted at 2026-08-06 10:47:44 on dist-test-slave-0b58"
server_key: "23d7d4783f44513720068cec599935ae"
server_key_iv: "5402b7116f4273a614dea77d349f6783"
server_key_version: "encryptionkey@0"
I20260806 10:47:44.444728 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "286aa4818a234c6c9c160433d46d8c58"
format_stamp: "Formatted at 2026-08-06 10:47:44 on dist-test-slave-0b58"
server_key: "23d7d4783f44513720068cec599935ae"
server_key_iv: "5402b7116f4273a614dea77d349f6783"
server_key_version: "encryptionkey@0"
I20260806 10:47:44.449702 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 10:47:44.453495 21745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:44.454743 21731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:44.454941 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "286aa4818a234c6c9c160433d46d8c58"
format_stamp: "Formatted at 2026-08-06 10:47:44 on dist-test-slave-0b58"
server_key: "23d7d4783f44513720068cec599935ae"
server_key_iv: "5402b7116f4273a614dea77d349f6783"
server_key_version: "encryptionkey@0"
I20260806 10:47:44.455161 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:44.477978 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:44.478847 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:44.479156 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:44.497866 21731 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35409
I20260806 10:47:44.497897 21798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35409 every 8 connection(s)
I20260806 10:47:44.499218 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:44.502743 21799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:44.509138 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21731
I20260806 10:47:44.509385 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:44.510030 16468 external_mini_cluster.cc:1428] Setting key 09fdfe52156e7b1d0a2ca6c673b31f84
I20260806 10:47:44.513230 21799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58: Bootstrap starting.
I20260806 10:47:44.516183 21799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:44.517266 21799 log.cc:824] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:44.519795 21799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58: No bootstrap required, opened a new log
I20260806 10:47:44.523013 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:44.511794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.16.21.62:35409)
0806 10:47:44.512750 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:44.512760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:44.512808 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:47:44.514569 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:44.514591 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:44.515006 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:44.515366 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:44.515920 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:44.517327 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:44.518694 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:44.519343 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:44.519563 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:44.520797 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:44.520805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:44.520818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:44.521352 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:44.521429 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:44.521453 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:44.521596 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:44.521839 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 10:47:44.522127 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":321,"threads_started":1}
I20260806 10:47:44.526675 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:47:44.527886 21799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286aa4818a234c6c9c160433d46d8c58" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35409 } }
I20260806 10:47:44.528234 21799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:44.528395 21799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286aa4818a234c6c9c160433d46d8c58, State: Initialized, Role: FOLLOWER
I20260806 10:47:44.528936 21799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286aa4818a234c6c9c160433d46d8c58" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35409 } }
I20260806 10:47:44.529238 21799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:44.529429 21799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:44.529660 21799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:44.531994 21799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286aa4818a234c6c9c160433d46d8c58" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35409 } }
I20260806 10:47:44.532486 21799 leader_election.cc:302] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 286aa4818a234c6c9c160433d46d8c58; no voters:
I20260806 10:47:44.532964 21799 leader_election.cc:288] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:44.533125 21804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:44.533970 21804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [term 1 LEADER]: Becoming Leader. State: Replica: 286aa4818a234c6c9c160433d46d8c58, State: Running, Role: LEADER
I20260806 10:47:44.534499 21804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286aa4818a234c6c9c160433d46d8c58" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35409 } }
I20260806 10:47:44.534961 21799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:44.537358 21806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286aa4818a234c6c9c160433d46d8c58. Latest consensus state: current_term: 1 leader_uuid: "286aa4818a234c6c9c160433d46d8c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286aa4818a234c6c9c160433d46d8c58" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35409 } } }
I20260806 10:47:44.537460 21805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "286aa4818a234c6c9c160433d46d8c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286aa4818a234c6c9c160433d46d8c58" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35409 } } }
I20260806 10:47:44.537786 21806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:44.537902 21805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:44.540014 21811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:44.546056 21811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:44.554806 21811 catalog_manager.cc:1347] Generated new cluster ID: 368d2b1551ab434abd8fdd8245a7ea40
I20260806 10:47:44.554919 21811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:44.562835 21811 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:44.563974 21811 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:44.573575 21811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 286aa4818a234c6c9c160433d46d8c58: Generated new TSK 0
I20260806 10:47:44.574331 21811 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 10:47:44.608507 21831 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 10:47:44.608721 21831 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 10:47:44.609862 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:44.589495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.16.21.62:35409)
0806 10:47:44.589683 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:44.589690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:44.590158 (+ 468us) server_negotiation.cc:407] Connection header received
0806 10:47:44.590529 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:44.590534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:44.590607 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:44.590739 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:47:44.591301 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:44.592035 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:44.593501 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:44.593991 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:44.594122 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:44.594655 (+ 533us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:47:44.609190 (+ 14535us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 10:47:44.609557 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 10:47:44.609985 21802 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 10:47:44.611431 21830 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:44.589441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35409 (local address 127.0.0.1:41680)
0806 10:47:44.590053 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:44.590068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:44.590262 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:44.590781 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:44.590787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:44.590802 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:47:44.591154 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:44.591162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:44.592184 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:44.592196 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:44.593354 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:44.593365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:44.594164 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:44.594173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:44.594345 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:44.594401 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:47:44.609448 (+ 15047us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 10:47:44.609755 (+ 307us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.21.62:35409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":333,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.21.62:35409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 10:47:44.613407 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21731
I20260806 10:47:44.614495 21794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:44.706547 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21731
2026-08-06T10:47:44Z chronyd exiting
I20260806 10:47:44.760551 16468 test_util.cc:175] -----------------------------------------------
I20260806 10:47:44.760687 16468 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013221924036-16468-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (735 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 10:47:44.764469 16468 mini_oidc.cc:152] Starting JWKS server
I20260806 10:47:44.766251 16468 webserver.cc:460] Webserver started at https://127.0.0.1:45415/ using document root <none> and password file <none>
I20260806 10:47:44.766448 16468 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 10:47:44.766992 16468 webserver.cc:460] Webserver started at http://127.0.0.1:36673/ using document root <none> and password file <none>
2026-08-06T10:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:44Z Disabled control of system clock
I20260806 10:47:44.803170 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:34831
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:34831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45415/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/testchainca.cert with env {}
W20260806 10:47:45.129904 21838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:45.130164 21838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:45.130367 21838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:45.137331 21838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:45.137446 21838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:45.137540 21838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:45.137645 21838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:45.137732 21838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:47:45.137821 21838 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:47:45.137877 21838 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45415/jwks.json
I20260806 10:47:45.147531 21838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:34831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45415/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:45.148823 21838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:45.149976 21838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:45.161710 21843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:45.161762 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:45.161796 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:45.162230 21838 server_base.cc:1034] running on GCE node
I20260806 10:47:45.163255 21838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:45.164669 21838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:45.165980 21838 hybrid_clock.cc:648] HybridClock initialized: now 1786013265165937 us; error 23 us; skew 500 ppm
I20260806 10:47:45.166369 21838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:45.172750 21838 webserver.cc:460] Webserver started at http://127.16.21.62:35495/ using document root <none> and password file <none>
I20260806 10:47:45.173395 21838 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:45.173528 21838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:45.173902 21838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:45.176753 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b"
format_stamp: "Formatted at 2026-08-06 10:47:45 on dist-test-slave-0b58"
server_key: "ddbf2b997745633baca584808628c6ed"
server_key_iv: "60999a50a32d8a2dfe8376cfbc23237c"
server_key_version: "encryptionkey@0"
I20260806 10:47:45.177615 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b"
format_stamp: "Formatted at 2026-08-06 10:47:45 on dist-test-slave-0b58"
server_key: "ddbf2b997745633baca584808628c6ed"
server_key_iv: "60999a50a32d8a2dfe8376cfbc23237c"
server_key_version: "encryptionkey@0"
I20260806 10:47:45.182390 21838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 10:47:45.185614 21853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:45.186830 21838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 10:47:45.187016 21838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b"
format_stamp: "Formatted at 2026-08-06 10:47:45 on dist-test-slave-0b58"
server_key: "ddbf2b997745633baca584808628c6ed"
server_key_iv: "60999a50a32d8a2dfe8376cfbc23237c"
server_key_version: "encryptionkey@0"
I20260806 10:47:45.187237 21838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:45.205502 21838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:45.206251 21838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:45.206558 21838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:45.226207 21838 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:34831
I20260806 10:47:45.226235 21905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:34831 every 8 connection(s)
I20260806 10:47:45.228016 21838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:45.232846 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:45.235195 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21838
I20260806 10:47:45.235391 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:45.236195 16468 external_mini_cluster.cc:1428] Setting key f79501b35d6f4911868faeaaac02ecc7
I20260806 10:47:45.249770 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b: Bootstrap starting.
I20260806 10:47:45.252583 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:45.238154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.16.21.62:34831)
0806 10:47:45.239280 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:45.239311 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:45.239370 (+ 59us) server_negotiation.cc:407] Connection header received
0806 10:47:45.241722 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:45.241756 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:45.242385 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:45.242816 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:45.243331 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.244892 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.246833 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.248014 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.248210 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:45.249599 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:45.249613 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:45.249633 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:45.250405 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:45.250533 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:45.250564 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:45.250687 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:45.250921 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 10:47:45.251229 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":418,"threads_started":1}
I20260806 10:47:45.253813 21906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:45.255311 21906 log.cc:824] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:45.256867 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:47:45.262148 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b: No bootstrap required, opened a new log
I20260806 10:47:45.270944 21906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34831 } }
I20260806 10:47:45.271309 21906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:45.271422 21906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efbebb762cd400b8c0fb9d31bd1bd1b, State: Initialized, Role: FOLLOWER
I20260806 10:47:45.272090 21906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34831 } }
I20260806 10:47:45.272369 21906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:45.272521 21906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:45.272710 21906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:45.275218 21906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34831 } }
I20260806 10:47:45.275745 21906 leader_election.cc:302] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1efbebb762cd400b8c0fb9d31bd1bd1b; no voters:
I20260806 10:47:45.276542 21906 leader_election.cc:288] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:45.276640 21911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:45.277405 21911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [term 1 LEADER]: Becoming Leader. State: Replica: 1efbebb762cd400b8c0fb9d31bd1bd1b, State: Running, Role: LEADER
I20260806 10:47:45.277904 21911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34831 } }
I20260806 10:47:45.278600 21906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:45.280884 21912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34831 } } }
I20260806 10:47:45.281351 21912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:45.282229 21913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efbebb762cd400b8c0fb9d31bd1bd1b. Latest consensus state: current_term: 1 leader_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1efbebb762cd400b8c0fb9d31bd1bd1b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34831 } } }
I20260806 10:47:45.282567 21913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:45.285143 21921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:45.289803 21921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:45.298985 21921 catalog_manager.cc:1347] Generated new cluster ID: e5d5c43f10b04531a6c139b456205e19
I20260806 10:47:45.299108 21921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:45.315873 21921 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:45.317052 21921 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:45.325825 21921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1efbebb762cd400b8c0fb9d31bd1bd1b: Generated new TSK 0
I20260806 10:47:45.326542 21921 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 10:47:45.364192 21939 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 10:47:45.364326 21939 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260806 10:47:45.365449 21938 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:45.346218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34831 (local address 127.0.0.1:41064)
0806 10:47:45.346884 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:45.346905 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:45.347016 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:45.347454 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:45.347458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:45.347467 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:47:45.347698 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:45.347706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.349188 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.349192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:45.350236 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:45.350247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.351232 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.351236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:45.351361 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:45.351390 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:47:45.365007 (+ 13617us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 10:47:45.365168 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.21.62:34831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":369,"threads_started":1}
I20260806 10:47:45.365490 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:45.346357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.16.21.62:34831)
0806 10:47:45.346535 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:45.346542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:45.346981 (+ 439us) server_negotiation.cc:407] Connection header received
0806 10:47:45.347178 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:45.347184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:45.347287 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:45.347428 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:47:45.347876 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.349059 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.350369 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.351059 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.351196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:45.351620 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:47:45.364813 (+ 13193us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 10:47:45.365182 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 10:47:45.365612 21909 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260806 10:47:45.366513 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21838
I20260806 10:47:45.367543 21901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:45.460285 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21838
2026-08-06T10:47:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 10:47:45.519766 16468 mini_oidc.cc:152] Starting JWKS server
I20260806 10:47:45.521730 16468 webserver.cc:460] Webserver started at https://127.0.0.1:38685/ using document root <none> and password file <none>
I20260806 10:47:45.521991 16468 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 10:47:45.522637 16468 webserver.cc:460] Webserver started at http://127.0.0.1:34431/ using document root <none> and password file <none>
2026-08-06T10:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:45Z Disabled control of system clock
I20260806 10:47:45.557044 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40363
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40363
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38685/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 10:47:45.881147 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:45.881487 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:45.881642 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:45.888963 21946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:45.889101 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:45.889190 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:45.889312 21946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:45.889390 21946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:47:45.889482 21946 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:47:45.889559 21946 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38685/jwks.json
I20260806 10:47:45.898794 21946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38685/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:45.900075 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:45.901978 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:45.913754 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:45.913810 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:45.913825 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:45.917553 21946 server_base.cc:1034] running on GCE node
I20260806 10:47:45.918315 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:45.919597 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:45.920791 21946 hybrid_clock.cc:648] HybridClock initialized: now 1786013265920753 us; error 26 us; skew 500 ppm
I20260806 10:47:45.921108 21946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:45.923243 21946 webserver.cc:460] Webserver started at http://127.16.21.62:46589/ using document root <none> and password file <none>
I20260806 10:47:45.923847 21946 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:45.923931 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:45.924211 21946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:45.927055 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "0ba286a0dbb949d0a5f7bfb672ff0293"
format_stamp: "Formatted at 2026-08-06 10:47:45 on dist-test-slave-0b58"
server_key: "b650940393c34a2e589b9bbf02309b43"
server_key_iv: "384d523e0d0ee709919c45e7124a4117"
server_key_version: "encryptionkey@0"
I20260806 10:47:45.927819 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "0ba286a0dbb949d0a5f7bfb672ff0293"
format_stamp: "Formatted at 2026-08-06 10:47:45 on dist-test-slave-0b58"
server_key: "b650940393c34a2e589b9bbf02309b43"
server_key_iv: "384d523e0d0ee709919c45e7124a4117"
server_key_version: "encryptionkey@0"
I20260806 10:47:45.932725 21946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:47:45.935887 21960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:45.937160 21946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:45.937309 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "0ba286a0dbb949d0a5f7bfb672ff0293"
format_stamp: "Formatted at 2026-08-06 10:47:45 on dist-test-slave-0b58"
server_key: "b650940393c34a2e589b9bbf02309b43"
server_key_iv: "384d523e0d0ee709919c45e7124a4117"
server_key_version: "encryptionkey@0"
I20260806 10:47:45.937481 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:45.953228 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:45.953902 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:45.954166 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:45.972754 21946 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40363
I20260806 10:47:45.972779 22011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40363 every 8 connection(s)
I20260806 10:47:45.974074 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:45.975967 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 21946
I20260806 10:47:45.976203 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:45.977114 16468 external_mini_cluster.cc:1428] Setting key 9c7abe29b9e9600472b1b195281ab169
I20260806 10:47:45.978484 22012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:45.991650 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:45.978957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.16.21.62:40363)
0806 10:47:45.979975 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:45.979991 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:45.980040 (+ 49us) server_negotiation.cc:407] Connection header received
0806 10:47:45.981892 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:45.981919 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:45.982486 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:45.982920 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:45.983416 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.985390 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.986704 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:45.987706 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:45.987903 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:45.988978 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:45.988989 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:45.989012 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:45.989708 (+ 696us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:45.989815 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:45.989840 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:45.989964 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:45.990118 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 10:47:45.990435 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
I20260806 10:47:45.993208 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293: Bootstrap starting.
I20260806 10:47:45.995298 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:47:45.996591 22012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:45.997785 22012 log.cc:824] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:46.000520 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293: No bootstrap required, opened a new log
I20260806 10:47:46.007377 22012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40363 } }
I20260806 10:47:46.007714 22012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:46.007859 22012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba286a0dbb949d0a5f7bfb672ff0293, State: Initialized, Role: FOLLOWER
I20260806 10:47:46.008486 22012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40363 } }
I20260806 10:47:46.008726 22012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:46.008863 22012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:46.009043 22012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:46.011065 22012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40363 } }
I20260806 10:47:46.011536 22012 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba286a0dbb949d0a5f7bfb672ff0293; no voters:
I20260806 10:47:46.012154 22012 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:46.012305 22017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:46.013041 22017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba286a0dbb949d0a5f7bfb672ff0293, State: Running, Role: LEADER
I20260806 10:47:46.013556 22017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40363 } }
I20260806 10:47:46.014237 22012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:46.016666 22018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40363 } } }
I20260806 10:47:46.017153 22018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:46.016957 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba286a0dbb949d0a5f7bfb672ff0293. Latest consensus state: current_term: 1 leader_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba286a0dbb949d0a5f7bfb672ff0293" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40363 } } }
I20260806 10:47:46.017308 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:46.018800 22026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:46.023743 22026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:46.032099 22026 catalog_manager.cc:1347] Generated new cluster ID: 9716756751a442889df7c1f07c2f7726
I20260806 10:47:46.032200 22026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:46.048524 22026 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:46.050232 22026 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:46.057178 22026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba286a0dbb949d0a5f7bfb672ff0293: Generated new TSK 0
I20260806 10:47:46.058036 22026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:47:46.080001 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:46.071635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.16.21.62:40363)
0806 10:47:46.071889 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:46.071895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:46.072250 (+ 355us) server_negotiation.cc:407] Connection header received
0806 10:47:46.072519 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:46.072523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:46.072587 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:46.072697 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:47:46.073305 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.074107 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.076030 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.076561 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.076651 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:46.077253 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:47:46.079403 (+ 2150us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 10:47:46.079705 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 10:47:46.080109 22015 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 10:47:46.082237 22041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:46.071509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40363 (local address 127.0.0.1:52136)
0806 10:47:46.072141 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:46.072161 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:46.072343 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:46.072750 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:46.072755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:46.072768 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:47:46.073096 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:46.073109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.074256 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.074262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:46.075893 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:46.075909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.076752 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.076759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:46.076968 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:46.077014 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:47:46.079615 (+ 2601us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 10:47:46.081732 (+ 2117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.21.62:40363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":1829,"thread_start_us":362,"threads_started":1}
I20260806 10:47:46.083258 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 21946
I20260806 10:47:46.084204 22007 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 10:47:46.092011 22042 webserver.cc:533] Webserver: sslize failed:
E20260806 10:47:46.092120 22042 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 10:47:46.173765 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 21946
2026-08-06T10:47:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T10:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:46Z Disabled control of system clock
I20260806 10:47:46.267010 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36711
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:36711
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/testchainca.cert with env {}
W20260806 10:47:46.598376 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:46.598685 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:46.598781 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:46.606077 22047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:46.606220 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:46.606269 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:46.606330 22047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:46.606369 22047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:47:46.606431 22047 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:47:46.606472 22047 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 10:47:46.616429 22047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:36711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:46.617700 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:46.619011 22047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:46.631016 22053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:46.631071 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:46.631328 22052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:46.631621 22047 server_base.cc:1034] running on GCE node
I20260806 10:47:46.632506 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:46.633769 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:46.634969 22047 hybrid_clock.cc:648] HybridClock initialized: now 1786013266634935 us; error 30 us; skew 500 ppm
I20260806 10:47:46.635324 22047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:47:46.637662 22047 webserver.cc:460] Webserver started at http://127.16.21.62:46869/ using document root <none> and password file <none>
I20260806 10:47:46.638342 22047 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:46.638458 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:46.638757 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:46.641625 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "85c8cddd9195452fbd4ce60d4b41c50a"
format_stamp: "Formatted at 2026-08-06 10:47:46 on dist-test-slave-0b58"
server_key: "516fb97354157e436ca1d77cd07d4fa5"
server_key_iv: "5766346a177eaa3667efc73313eae0c1"
server_key_version: "encryptionkey@0"
I20260806 10:47:46.642427 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "85c8cddd9195452fbd4ce60d4b41c50a"
format_stamp: "Formatted at 2026-08-06 10:47:46 on dist-test-slave-0b58"
server_key: "516fb97354157e436ca1d77cd07d4fa5"
server_key_iv: "5766346a177eaa3667efc73313eae0c1"
server_key_version: "encryptionkey@0"
I20260806 10:47:46.647506 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:47:46.650761 22061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:46.652055 22047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:46.652241 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "85c8cddd9195452fbd4ce60d4b41c50a"
format_stamp: "Formatted at 2026-08-06 10:47:46 on dist-test-slave-0b58"
server_key: "516fb97354157e436ca1d77cd07d4fa5"
server_key_iv: "5766346a177eaa3667efc73313eae0c1"
server_key_version: "encryptionkey@0"
I20260806 10:47:46.652493 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:46.668515 22047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:46.669277 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:46.669552 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:46.688776 22047 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:36711
I20260806 10:47:46.688803 22112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:36711 every 8 connection(s)
I20260806 10:47:46.690024 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:46.693545 22113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:46.697474 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22047
I20260806 10:47:46.697701 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:46.698372 16468 external_mini_cluster.cc:1428] Setting key 7b4593597e3f5469468bfd56fa57658f
I20260806 10:47:46.704113 22113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a: Bootstrap starting.
I20260806 10:47:46.707067 22113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:46.708102 22113 log.cc:824] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:46.711581 22113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a: No bootstrap required, opened a new log
I20260806 10:47:46.711982 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:46.700273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.16.21.62:36711)
0806 10:47:46.701215 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:46.701225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:46.701278 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:47:46.703022 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:46.703044 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:46.703450 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:46.703803 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:46.704354 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.705650 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.706923 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.707667 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.707861 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:46.709264 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:46.709272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:46.709287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:47:46.709970 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:46.710068 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:47:46.710094 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:46.710219 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:46.710423 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 10:47:46.710741 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":350,"threads_started":1}
I20260806 10:47:46.715796 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:47:46.720894 22113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36711 } }
I20260806 10:47:46.721201 22113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:46.721285 22113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c8cddd9195452fbd4ce60d4b41c50a, State: Initialized, Role: FOLLOWER
I20260806 10:47:46.721798 22113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36711 } }
I20260806 10:47:46.721993 22113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:46.722119 22113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:46.722278 22113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:46.724732 22113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36711 } }
I20260806 10:47:46.725147 22113 leader_election.cc:302] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c8cddd9195452fbd4ce60d4b41c50a; no voters:
I20260806 10:47:46.725687 22113 leader_election.cc:288] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:46.725847 22118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:46.726584 22118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [term 1 LEADER]: Becoming Leader. State: Replica: 85c8cddd9195452fbd4ce60d4b41c50a, State: Running, Role: LEADER
I20260806 10:47:46.727116 22118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36711 } }
I20260806 10:47:46.727655 22113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:46.730132 22120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c8cddd9195452fbd4ce60d4b41c50a. Latest consensus state: current_term: 1 leader_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36711 } } }
I20260806 10:47:46.731210 22120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:46.730908 22119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c8cddd9195452fbd4ce60d4b41c50a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36711 } } }
I20260806 10:47:46.731395 22119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:46.738945 22123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:46.744050 22123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:46.759294 22123 catalog_manager.cc:1347] Generated new cluster ID: d021e8e38a664d4dbe5ab4633f607f96
I20260806 10:47:46.759418 22123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:46.784366 22123 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:46.785594 22123 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:46.797351 22123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c8cddd9195452fbd4ce60d4b41c50a: Generated new TSK 0
I20260806 10:47:46.798149 22123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:47:47.243613 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:46.811932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.16.21.62:36711)
0806 10:47:46.812127 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:46.812134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:46.812581 (+ 447us) server_negotiation.cc:407] Connection header received
0806 10:47:46.812788 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:46.812796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:46.812893 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:46.813049 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:47:46.813469 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.814620 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.816007 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.816805 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.816935 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:46.817532 (+ 597us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:47:47.243035 (+425503us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 10:47:47.243387 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51002: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 10:47:47.243770 22116 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51002: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 10:47:47.243968 22144 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:46.811780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36711 (local address 127.0.0.1:51002)
0806 10:47:46.812482 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:46.812497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:46.812614 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:46.813070 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:46.813073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:46.813082 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:47:46.813322 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:46.813330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.814777 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.814781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:46.815865 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:46.815877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:46.817018 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:46.817023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:46.817196 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:46.817231 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:47:47.243325 (+426094us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 10:47:47.243577 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.21.62:36711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":393,"threads_started":1}
I20260806 10:47:47.245340 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22047
I20260806 10:47:47.246419 22108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:47.355724 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22047
2026-08-06T10:47:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1165 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (420 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (424 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:48 dist-test-slave-0b58 krb5kdc[22159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:48 dist-test-slave-0b58 krb5kdc[22159](info): set up 2 sockets
Aug 06 10:47:48 dist-test-slave-0b58 krb5kdc[22159](info): commencing operation
krb5kdc: starting...
W20260806 10:47:49.313555 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:49 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:47:49.544015 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38327
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:49.868017 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:49.868288 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:49.868377 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:49.875223 22172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:49.875310 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:49.875365 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:49.875413 22172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:49.875458 22172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:49.884644 22172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:38327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:49.885701 22172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:49.886879 22172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:49.898022 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:49.897990 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:49.898052 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:49.899819 22172 server_base.cc:1034] running on GCE node
I20260806 10:47:49.900570 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:49.901777 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:49.903019 22172 hybrid_clock.cc:648] HybridClock initialized: now 1786013269902984 us; error 45 us; skew 500 ppm
I20260806 10:47:49.903362 22172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:49 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:49.906452 22172 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:47:49.907858 22172 webserver.cc:460] Webserver started at http://127.16.21.62:34967/ using document root <none> and password file <none>
I20260806 10:47:49.908556 22172 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:49.908658 22172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:49.908921 22172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:49.911660 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "49e31cd6196547ac906481529666e587"
format_stamp: "Formatted at 2026-08-06 10:47:49 on dist-test-slave-0b58"
server_key: "bf1c0b7af14d661c7450f20724460aed"
server_key_iv: "23c1f1fe9e113cc5983f62571010ae70"
server_key_version: "encryptionkey@0"
I20260806 10:47:49.912442 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "49e31cd6196547ac906481529666e587"
format_stamp: "Formatted at 2026-08-06 10:47:49 on dist-test-slave-0b58"
server_key: "bf1c0b7af14d661c7450f20724460aed"
server_key_iv: "23c1f1fe9e113cc5983f62571010ae70"
server_key_version: "encryptionkey@0"
I20260806 10:47:49.917408 22172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:49.920641 22188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:49.921875 22172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:47:49.922049 22172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "49e31cd6196547ac906481529666e587"
format_stamp: "Formatted at 2026-08-06 10:47:49 on dist-test-slave-0b58"
server_key: "bf1c0b7af14d661c7450f20724460aed"
server_key_iv: "23c1f1fe9e113cc5983f62571010ae70"
server_key_version: "encryptionkey@0"
I20260806 10:47:49.922240 22172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:49.941222 22172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:49.944962 22172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:49.945256 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:49.964044 22172 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:38327
I20260806 10:47:49.964090 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:38327 every 8 connection(s)
I20260806 10:47:49.965324 22172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:49.968901 22241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:49.975872 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22172
I20260806 10:47:49.976091 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:49.976944 16468 external_mini_cluster.cc:1428] Setting key 95362150db674c365e7ad82d0e6c20c7
I20260806 10:47:49.979779 22241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587: Bootstrap starting.
I20260806 10:47:49.983078 22241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:49.984391 22241 log.cc:824] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:49.987498 22241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587: No bootstrap required, opened a new log
Aug 06 10:47:49 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:49.996353 22241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e31cd6196547ac906481529666e587" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38327 } }
I20260806 10:47:49.996794 22241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:49.996922 22241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e31cd6196547ac906481529666e587, State: Initialized, Role: FOLLOWER
I20260806 10:47:49.997550 22241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e31cd6196547ac906481529666e587" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38327 } }
I20260806 10:47:49.997829 22241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:49.997970 22241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:49.998193 22241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:50.001123 22241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e31cd6196547ac906481529666e587" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38327 } }
I20260806 10:47:50.001646 22241 leader_election.cc:302] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49e31cd6196547ac906481529666e587; no voters:
I20260806 10:47:50.001935 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:49.979208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.16.21.62:38327)
0806 10:47:49.980082 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:49.980093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:49.980132 (+ 39us) server_negotiation.cc:407] Connection header received
0806 10:47:49.982128 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:49.982149 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:49.982498 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:49.982802 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:49.984703 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:49.986479 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:49.988351 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:49.988990 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:49.989135 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:49.993834 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:49.993843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:49.993857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:49.993893 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:49.997194 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:49.998109 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:49.998117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:49.998126 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:49.998389 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:49.998959 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:49.998963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:49.998965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:49.999539 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:49.999775 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:50.000416 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 10:47:50.000832 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":447,"threads_started":1}
I20260806 10:47:50.002343 22241 leader_election.cc:288] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:50.002449 22246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:50.003834 22246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [term 1 LEADER]: Becoming Leader. State: Replica: 49e31cd6196547ac906481529666e587, State: Running, Role: LEADER
I20260806 10:47:50.004458 22246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e31cd6196547ac906481529666e587" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38327 } }
I20260806 10:47:50.004874 22241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:50.007403 22248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49e31cd6196547ac906481529666e587. Latest consensus state: current_term: 1 leader_uuid: "49e31cd6196547ac906481529666e587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e31cd6196547ac906481529666e587" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38327 } } }
I20260806 10:47:50.007808 22248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:50.008342 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49e31cd6196547ac906481529666e587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e31cd6196547ac906481529666e587" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38327 } } }
I20260806 10:47:50.009029 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:50.017196 22253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:50.022037 22253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:50.031337 22253 catalog_manager.cc:1347] Generated new cluster ID: 7a68ca9b21fa4d598f5afe8cff7e9b8c
I20260806 10:47:50.031450 22253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:50.046515 22253 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:50.048278 22253 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:50.065938 22253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49e31cd6196547ac906481529666e587: Generated new TSK 0
I20260806 10:47:50.066754 22253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:47:50.129174 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38327
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:50.466581 22267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:50.466898 22267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:50.467087 22267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:50.474351 22267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:50.474500 22267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:50.474681 22267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:50.484364 22267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:50.485600 22267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:50.487129 22267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:50.499101 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:50.499858 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:50.500495 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:50.501487 22267 server_base.cc:1034] running on GCE node
I20260806 10:47:50.502113 22267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:50.503000 22267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:50.504204 22267 hybrid_clock.cc:648] HybridClock initialized: now 1786013270504163 us; error 36 us; skew 500 ppm
I20260806 10:47:50.504565 22267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:50 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013270, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:50.510743 22267 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:47:50.512238 22267 webserver.cc:460] Webserver started at http://127.16.21.1:39429/ using document root <none> and password file <none>
I20260806 10:47:50.512941 22267 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:50.513041 22267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:50.513293 22267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:50.516126 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "c701f6cc2b504b9bb2c0fca06583897f"
format_stamp: "Formatted at 2026-08-06 10:47:50 on dist-test-slave-0b58"
server_key: "a19b2be4f0bf03667342f03e00579d09"
server_key_iv: "3cba158d2fffd806f1027647f72c4d4e"
server_key_version: "encryptionkey@0"
I20260806 10:47:50.516975 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "c701f6cc2b504b9bb2c0fca06583897f"
format_stamp: "Formatted at 2026-08-06 10:47:50 on dist-test-slave-0b58"
server_key: "a19b2be4f0bf03667342f03e00579d09"
server_key_iv: "3cba158d2fffd806f1027647f72c4d4e"
server_key_version: "encryptionkey@0"
I20260806 10:47:50.522198 22267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:47:50.525672 22282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:50.527060 22267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:50.527256 22267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "c701f6cc2b504b9bb2c0fca06583897f"
format_stamp: "Formatted at 2026-08-06 10:47:50 on dist-test-slave-0b58"
server_key: "a19b2be4f0bf03667342f03e00579d09"
server_key_iv: "3cba158d2fffd806f1027647f72c4d4e"
server_key_version: "encryptionkey@0"
I20260806 10:47:50.527467 22267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:50.562739 22267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:50.567018 22267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:50.567328 22267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:50.568583 22267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:50.569960 22267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:50.570035 22267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:50.570214 22267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:50.570277 22267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:50.604007 22267 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:38175
I20260806 10:47:50.604094 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:38175 every 8 connection(s)
I20260806 10:47:50.605423 22267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:50.612201 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22267
I20260806 10:47:50.612455 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:50.613360 16468 external_mini_cluster.cc:1428] Setting key 8bb101ceda95294c5968da142a7db723
Aug 06 10:47:50 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013270, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:50.627316 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:50.607978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:35687 (local address 127.16.21.62:38327)
0806 10:47:50.608493 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:50.608497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:50.609416 (+ 919us) server_negotiation.cc:407] Connection header received
0806 10:47:50.611244 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:50.611249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:50.611349 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:50.611468 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:50.613560 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:50.614441 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:50.616681 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:50.617070 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:50.617168 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:50.621399 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:50.621403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:50.621407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:50.621436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:50.624115 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:50.624899 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:50.624904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:50.624906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:50.625024 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:50.625505 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:50.625510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:50.625515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:50.625817 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:50.625983 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:50.626682 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 10:47:50.626938 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":295,"threads_started":1}
I20260806 10:47:50.628139 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:50.608527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38327 (local address 127.16.21.1:35687)
0806 10:47:50.609257 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:50.609302 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:50.610983 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:50.611727 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:50.611740 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:50.612559 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:50.613374 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:50.613405 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:50.614607 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:50.614612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:50.616536 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:50.616556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:50.617306 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:50.617316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:50.617646 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:50.618408 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:50.618450 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:50.621166 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:50.624317 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:50.624325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:50.624339 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:50.624761 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:50.625138 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:50.625143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:50.625147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:50.625376 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:50.625954 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:50.625961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:50.626290 (+ 329us) client_negotiation.cc:769] Sending connection context
0806 10:47:50.626555 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 10:47:50.627022 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":327,"threads_started":1}
I20260806 10:47:50.629786 22396 heartbeater.cc:347] Connected to a master server at 127.16.21.62:38327
I20260806 10:47:50.630028 22396 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:50.630648 22396 heartbeater.cc:511] Master 127.16.21.62:38327 requested a full tablet report, sending...
I20260806 10:47:50.632951 22205 ts_manager.cc:194] Registered new tserver with Master: c701f6cc2b504b9bb2c0fca06583897f (127.16.21.1:38175)
I20260806 10:47:50.635110 22205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:35687
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:47:50.671099 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38327
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:50.991904 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:50.992134 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:50.992230 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:50.999238 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:50.999336 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:50.999437 22402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:51.008767 22402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:51.009850 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:51.011059 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:51.023530 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:51.023564 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:51.024370 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:51.024670 22402 server_base.cc:1034] running on GCE node
I20260806 10:47:51.025242 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:51.026175 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:51.027392 22402 hybrid_clock.cc:648] HybridClock initialized: now 1786013271027340 us; error 33 us; skew 500 ppm
I20260806 10:47:51.027694 22402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.030750 22402 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:47:51.032140 22402 webserver.cc:460] Webserver started at http://127.16.21.2:37923/ using document root <none> and password file <none>
I20260806 10:47:51.032859 22402 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:51.032961 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:51.033202 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:51.035866 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "e9920b258dd74b32ac2bca406805fa09"
format_stamp: "Formatted at 2026-08-06 10:47:51 on dist-test-slave-0b58"
server_key: "358b8b76fa42735c8174bec510725e2f"
server_key_iv: "71c502af0d49a89190c7d641aed864d1"
server_key_version: "encryptionkey@0"
I20260806 10:47:51.036648 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "e9920b258dd74b32ac2bca406805fa09"
format_stamp: "Formatted at 2026-08-06 10:47:51 on dist-test-slave-0b58"
server_key: "358b8b76fa42735c8174bec510725e2f"
server_key_iv: "71c502af0d49a89190c7d641aed864d1"
server_key_version: "encryptionkey@0"
I20260806 10:47:51.041371 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 10:47:51.044662 22417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:51.045861 22402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:47:51.046033 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "e9920b258dd74b32ac2bca406805fa09"
format_stamp: "Formatted at 2026-08-06 10:47:51 on dist-test-slave-0b58"
server_key: "358b8b76fa42735c8174bec510725e2f"
server_key_iv: "71c502af0d49a89190c7d641aed864d1"
server_key_version: "encryptionkey@0"
I20260806 10:47:51.046248 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:51.061390 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:51.064960 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:51.065238 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:51.066102 22402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:51.067395 22402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:51.067499 22402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:51.067659 22402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:51.067734 22402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:51.100950 22402 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:46639
I20260806 10:47:51.100973 22532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:46639 every 8 connection(s)
I20260806 10:47:51.102193 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:51.111490 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22402
I20260806 10:47:51.111666 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:51.112308 16468 external_mini_cluster.cc:1428] Setting key 1fa1a15cd0685976ab5e94ef3a587405
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:51.123646 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.104679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:47983 (local address 127.16.21.62:38327)
0806 10:47:51.104857 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.104862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.106022 (+ 1160us) server_negotiation.cc:407] Connection header received
0806 10:47:51.107885 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.107891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.107982 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.108094 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.109573 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.110287 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.112356 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.112884 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.112990 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.116654 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.116662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.116684 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.116731 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.120486 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.121233 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.121241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.121247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.121411 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.121815 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.121822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.121827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.122192 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.122354 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.122934 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.123214 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:47:51.124185 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.105066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38327 (local address 127.16.21.2:47983)
0806 10:47:51.105860 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:51.105909 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:51.107612 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:51.108364 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:51.108372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:51.108879 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:51.109404 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:51.109420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.110459 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.110463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:51.112157 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:51.112172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.113129 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.113136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:51.113398 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.114078 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:51.114136 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:51.116408 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:51.120620 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:51.120626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:51.120636 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:51.121111 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:51.121523 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:51.121526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:51.121528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:51.121695 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:51.122312 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:51.122318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:51.122584 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 10:47:51.122789 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 10:47:51.123121 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":372,"threads_started":1}
I20260806 10:47:51.125643 22534 heartbeater.cc:347] Connected to a master server at 127.16.21.62:38327
I20260806 10:47:51.125947 22534 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:51.126780 22534 heartbeater.cc:511] Master 127.16.21.62:38327 requested a full tablet report, sending...
I20260806 10:47:51.128192 22205 ts_manager.cc:194] Registered new tserver with Master: e9920b258dd74b32ac2bca406805fa09 (127.16.21.2:46639)
I20260806 10:47:51.129027 22205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:47983
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:47:51.167022 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38327
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:51.488819 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:51.489110 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:51.489290 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:51.496484 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:51.496614 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:51.496779 22539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:51.506225 22539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:51.507413 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:51.508721 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:51.521970 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:51.521793 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:51.522087 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:51.522954 22539 server_base.cc:1034] running on GCE node
I20260806 10:47:51.523509 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:51.524428 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:51.525645 22539 hybrid_clock.cc:648] HybridClock initialized: now 1786013271525608 us; error 26 us; skew 500 ppm
I20260806 10:47:51.525949 22539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.529356 22539 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:47:51.530704 22539 webserver.cc:460] Webserver started at http://127.16.21.3:39027/ using document root <none> and password file <none>
I20260806 10:47:51.531409 22539 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:51.531558 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:51.531855 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:51.534659 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "0db4342974d7460ebc666f9ef9f01a29"
format_stamp: "Formatted at 2026-08-06 10:47:51 on dist-test-slave-0b58"
server_key: "ab172d2586d261cf8668c22ceaf1d414"
server_key_iv: "1b8d3cd96e58f56fe1d67fa805105f5c"
server_key_version: "encryptionkey@0"
I20260806 10:47:51.535413 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "0db4342974d7460ebc666f9ef9f01a29"
format_stamp: "Formatted at 2026-08-06 10:47:51 on dist-test-slave-0b58"
server_key: "ab172d2586d261cf8668c22ceaf1d414"
server_key_iv: "1b8d3cd96e58f56fe1d67fa805105f5c"
server_key_version: "encryptionkey@0"
I20260806 10:47:51.540099 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 10:47:51.543252 22554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:51.544602 22539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:51.544771 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "0db4342974d7460ebc666f9ef9f01a29"
format_stamp: "Formatted at 2026-08-06 10:47:51 on dist-test-slave-0b58"
server_key: "ab172d2586d261cf8668c22ceaf1d414"
server_key_iv: "1b8d3cd96e58f56fe1d67fa805105f5c"
server_key_version: "encryptionkey@0"
I20260806 10:47:51.544965 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:51.575065 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:51.579492 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:51.579799 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:51.580749 22539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:51.582074 22539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:51.582186 22539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:51.582345 22539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:51.582428 22539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:51.616190 22539 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:44553
I20260806 10:47:51.616310 22667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:44553 every 8 connection(s)
I20260806 10:47:51.617436 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:51.618362 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22539
I20260806 10:47:51.618556 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:51.619100 16468 external_mini_cluster.cc:1428] Setting key 813d070facf84be5ac42e806c0dbfe3e
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:47:51.637080 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.620164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:59949 (local address 127.16.21.62:38327)
0806 10:47:51.620351 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.620355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.621810 (+ 1455us) server_negotiation.cc:407] Connection header received
0806 10:47:51.623700 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.623704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.623788 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.623866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.625395 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.626141 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.627741 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.628153 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.628243 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.631507 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.631511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.631514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.631545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.634469 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.635162 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.635166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.635168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.635300 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.635656 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.635659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.635661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.635894 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.636016 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.636658 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.636807 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 10:47:51.637867 22396 heartbeater.cc:503] Master 127.16.21.62:38327 was elected leader, sending a full tablet report...
I20260806 10:47:51.637894 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.620712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38327 (local address 127.16.21.3:59949)
0806 10:47:51.621641 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:51.621687 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:51.623478 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:51.624062 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:51.624069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:51.624634 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:51.625185 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:51.625211 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.626351 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.626356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:51.627611 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:51.627623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.628382 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.628388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:51.628570 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.629163 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:51.629184 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:51.631322 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:51.634594 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:51.634603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:51.634620 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:51.635055 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:51.635390 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:51.635393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:51.635395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:51.635562 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:51.636012 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:51.636017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:51.636349 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 10:47:51.636556 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 10:47:51.636887 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":396,"threads_started":1}
I20260806 10:47:51.639304 22668 heartbeater.cc:347] Connected to a master server at 127.16.21.62:38327
I20260806 10:47:51.639624 22668 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:51.640352 22668 heartbeater.cc:511] Master 127.16.21.62:38327 requested a full tablet report, sending...
I20260806 10:47:51.641623 22205 ts_manager.cc:194] Registered new tserver with Master: 0db4342974d7460ebc666f9ef9f01a29 (127.16.21.3:44553)
I20260806 10:47:51.642409 22205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:59949
I20260806 10:47:51.646541 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/test-admin.keytab.
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.679162 16468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 10:47:51.684799 22675 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:51.682045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.16.21.1:38175)
0806 10:47:51.682742 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.682749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.682775 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:47:51.682839 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.682845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.683066 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.683275 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.684335 (+ 1060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36978: BlockingRecv error: recv got EOF from 127.0.0.1:36978 (error 108)
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":342,"threads_started":1}
W20260806 10:47:51.684893 16468 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc' of type FILE
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.686784 16468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.688210 16468 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:51.704123 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.689993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.16.21.1:38175)
0806 10:47:51.690178 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.690184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.690334 (+ 150us) server_negotiation.cc:407] Connection header received
0806 10:47:51.690470 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.690473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.690528 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.690606 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.691607 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.692815 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.694097 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.694695 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.694799 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.698487 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.698498 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.698513 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.698545 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.701316 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.702124 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.702130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.702132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.702256 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.702605 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.702608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.702610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.702969 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.703198 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.703546 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.703849 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 10:47:51.710026 22675 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:51.708204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.16.21.1:38175)
0806 10:47:51.708452 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.708457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.708713 (+ 256us) server_negotiation.cc:407] Connection header received
0806 10:47:51.708884 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.708889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.708954 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.709056 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.709765 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37000: BlockingRecv error: recv got EOF from 127.0.0.1:37000 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260806 10:47:51.710534 16468 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc' of type FILE
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.711973 16468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.713438 16468 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:51.727466 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.714851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.16.21.1:38175)
0806 10:47:51.715045 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.715051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.715223 (+ 172us) server_negotiation.cc:407] Connection header received
0806 10:47:51.715400 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.715409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.715462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.715559 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.716586 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.717304 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.718515 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.718928 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.719064 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.722722 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.722727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.722731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.722762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.725146 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.725796 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.725803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.725808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.725968 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.726312 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.726315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.726317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.726620 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.726818 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.727092 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.727240 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 10:47:51.731874 22675 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:47:51.729943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.16.21.1:38175)
0806 10:47:51.730164 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.730170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.730392 (+ 222us) server_negotiation.cc:407] Connection header received
0806 10:47:51.730565 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.730570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.730645 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.730742 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.731645 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37030: BlockingRecv error: recv got EOF from 127.0.0.1:37030 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 10:47:51.732021 16468 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013221924036-16468-0/krb5kdc/krb5cc' of type FILE
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.733532 16468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:51.734758 16468 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:47:51.747627 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.735978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.16.21.1:38175)
0806 10:47:51.736119 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.736123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.736552 (+ 429us) server_negotiation.cc:407] Connection header received
0806 10:47:51.736645 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.736648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.736694 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.736772 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.737651 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.738342 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.739478 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.739883 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.739969 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.743175 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.743179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.743182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.743209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.745527 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.746080 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.746083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.746085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.746231 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.746529 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.746532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.746534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.746765 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.746995 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.747122 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.747316 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:47:51.749048 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22267
I20260806 10:47:51.759871 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.749842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.16.21.1:38175)
0806 10:47:51.749977 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.749980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.750304 (+ 324us) server_negotiation.cc:407] Connection header received
0806 10:47:51.750407 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.750410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.750455 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.750525 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.751380 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.752000 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.753162 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.753576 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.753667 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.755453 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.755457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.755461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.755487 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.757765 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.758362 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.758365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.758367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.758491 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.758794 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.758797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.758799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.759014 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.759221 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.759494 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.759637 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 10:47:51.760468 22391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:51.843724 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22267
I20260806 10:47:51.863405 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22402
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:47:51.878101 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.864338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.16.21.2:46639)
0806 10:47:51.864883 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.864890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.864916 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:47:51.864977 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.864983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.865186 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.865319 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.866305 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.867312 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.868599 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.869150 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.869245 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.872864 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.872875 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.872887 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.872915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.875470 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.876089 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.876095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.876098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.876210 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.876716 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.876719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.876721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.877039 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.877260 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.877532 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.877796 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":251,"threads_started":1}
I20260806 10:47:51.879379 22528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:51.961778 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22402
I20260806 10:47:51.980500 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22539
Aug 06 10:47:51 dist-test-slave-0b58 krb5kdc[22159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:47:51.996062 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:51.981492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.16.21.3:44553)
0806 10:47:51.982085 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:51.982092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:51.982121 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:47:51.982177 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:51.982183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:51.982397 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:51.982538 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:51.983520 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.984637 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.985948 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:51.986505 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:51.986630 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:51.990610 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:51.990620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:51.990634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:51.990665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:51.993330 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.993994 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.994003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.994006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.994124 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:51.994522 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:51.994528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:51.994531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:51.994879 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:51.995195 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:51.995476 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 10:47:51.995751 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":306,"threads_started":1}
I20260806 10:47:51.997345 22663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:52.079596 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22539
I20260806 10:47:52.098394 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22172
I20260806 10:47:52.099555 22236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:52.203161 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22172
2026-08-06T10:47:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4008 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:47:52 dist-test-slave-0b58 krb5kdc[22696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:47:52 dist-test-slave-0b58 krb5kdc[22696](info): set up 2 sockets
Aug 06 10:47:52 dist-test-slave-0b58 krb5kdc[22696](info): commencing operation
krb5kdc: starting...
W20260806 10:47:53.333520 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:47:53 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:47:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab.
Entry for principal oryx/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab.
I20260806 10:47:53.580427 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45475
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab
--principal=oryx/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:53.914317 22709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:53.914634 22709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:53.914814 22709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:53.922434 22709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:53.922586 22709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:53.922674 22709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:53.922757 22709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:53.922842 22709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:53.932384 22709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab
--principal=oryx/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:53.933673 22709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:53.935125 22709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:53.946832 22715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:53.946918 22717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:53.946888 22714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:53.947762 22709 server_base.cc:1034] running on GCE node
I20260806 10:47:53.948509 22709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:53.949841 22709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:53.951035 22709 hybrid_clock.cc:648] HybridClock initialized: now 1786013273950979 us; error 49 us; skew 500 ppm
I20260806 10:47:53.951406 22709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:53 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:53.954427 22709 init.cc:377] Logged in from keytab as oryx/127.16.21.62@KRBTEST.COM (short username oryx)
I20260806 10:47:53.955858 22709 webserver.cc:460] Webserver started at http://127.16.21.62:36915/ using document root <none> and password file <none>
I20260806 10:47:53.956619 22709 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:53.956771 22709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:53.957082 22709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:53.960007 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "ece5d271795e4851b5e02b6f915b0710"
format_stamp: "Formatted at 2026-08-06 10:47:53 on dist-test-slave-0b58"
server_key: "13e53654e2f9f36b117ca17ab72e52e9"
server_key_iv: "ac9ad845742d21fa0c46c3a2bff74d8f"
server_key_version: "encryptionkey@0"
I20260806 10:47:53.960819 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "ece5d271795e4851b5e02b6f915b0710"
format_stamp: "Formatted at 2026-08-06 10:47:53 on dist-test-slave-0b58"
server_key: "13e53654e2f9f36b117ca17ab72e52e9"
server_key_iv: "ac9ad845742d21fa0c46c3a2bff74d8f"
server_key_version: "encryptionkey@0"
I20260806 10:47:53.965577 22709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 10:47:53.968787 22724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:53.970113 22709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:47:53.970296 22709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "ece5d271795e4851b5e02b6f915b0710"
format_stamp: "Formatted at 2026-08-06 10:47:53 on dist-test-slave-0b58"
server_key: "13e53654e2f9f36b117ca17ab72e52e9"
server_key_iv: "ac9ad845742d21fa0c46c3a2bff74d8f"
server_key_version: "encryptionkey@0"
I20260806 10:47:53.970484 22709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:53.990232 22709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:53.994158 22709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:53.994452 22709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:54.013401 22709 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:45475
I20260806 10:47:54.013446 22775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:45475 every 8 connection(s)
I20260806 10:47:54.014757 22709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:47:54.018348 22776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:54.024230 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22709
I20260806 10:47:54.024494 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:47:54.025130 16468 external_mini_cluster.cc:1428] Setting key 39cf1c7ec8d3d9413b568b509d0478c3
I20260806 10:47:54.029431 22776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Bootstrap starting.
I20260806 10:47:54.033284 22776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:54.034557 22776 log.cc:824] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:54.037778 22776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: No bootstrap required, opened a new log
Aug 06 10:47:54 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:47:54.046136 22776 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } }
I20260806 10:47:54.046571 22776 raft_consensus.cc:374] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:54.046689 22776 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece5d271795e4851b5e02b6f915b0710, State: Initialized, Role: FOLLOWER
I20260806 10:47:54.047374 22776 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } }
I20260806 10:47:54.047684 22776 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:47:54.047850 22776 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:47:54.048041 22776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:54.049615 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:54.027133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.16.21.62:45475)
0806 10:47:54.027966 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:54.027983 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:54.028033 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:47:54.030491 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:54.030518 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:54.030966 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:54.031451 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:54.032851 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:54.034909 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:54.036542 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:54.037575 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:54.037752 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:54.042031 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:54.042042 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:54.042057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:54.042103 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:54.045389 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:54.046206 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:54.046212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:54.046217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:54.046387 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:54.046810 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:54.046815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:54.046819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:54.047359 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:54.047642 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:54.048144 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 10:47:54.048629 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":338,"threads_started":1}
I20260806 10:47:54.050580 22776 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } }
I20260806 10:47:54.051139 22776 leader_election.cc:302] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece5d271795e4851b5e02b6f915b0710; no voters:
I20260806 10:47:54.051882 22776 leader_election.cc:288] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:47:54.052000 22781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:54.053385 22781 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 LEADER]: Becoming Leader. State: Replica: ece5d271795e4851b5e02b6f915b0710, State: Running, Role: LEADER
I20260806 10:47:54.053886 22781 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } }
I20260806 10:47:54.054025 22776 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:54.056676 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece5d271795e4851b5e02b6f915b0710. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } }
I20260806 10:47:54.056687 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } }
I20260806 10:47:54.057219 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:54.057219 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:54.061756 22788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:47:54.066723 22788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:47:54.074957 22788 catalog_manager.cc:1347] Generated new cluster ID: 2a75c74a8c5b42cdaa63240d0a22fb29
I20260806 10:47:54.075062 22788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:47:54.087301 22788 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:47:54.088603 22788 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:47:54.095715 22788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Generated new TSK 0
I20260806 10:47:54.096457 22788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab.
Entry for principal oryx/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab.
I20260806 10:47:54.156301 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45475
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:54.483225 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:54.483461 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:54.483546 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:54.490553 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:54.490640 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:54.490793 22802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:54.500140 22802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:54.501235 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:54.502411 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:54.514981 22807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:54.515044 22810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:54.515120 22808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:54.516031 22802 server_base.cc:1034] running on GCE node
I20260806 10:47:54.516633 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:54.517470 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:54.518663 22802 hybrid_clock.cc:648] HybridClock initialized: now 1786013274518614 us; error 39 us; skew 500 ppm
I20260806 10:47:54.518981 22802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:54 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013274, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:54.521875 22802 init.cc:377] Logged in from keytab as oryx/127.16.21.1@KRBTEST.COM (short username oryx)
I20260806 10:47:54.523167 22802 webserver.cc:460] Webserver started at http://127.16.21.1:37719/ using document root <none> and password file <none>
I20260806 10:47:54.523799 22802 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:54.523942 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:54.524226 22802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:54.526993 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce"
format_stamp: "Formatted at 2026-08-06 10:47:54 on dist-test-slave-0b58"
server_key: "eb80e4ccae5bc0aa49ba99f155664881"
server_key_iv: "757c19d060f2c4d57ef699cb3df765a2"
server_key_version: "encryptionkey@0"
I20260806 10:47:54.527788 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce"
format_stamp: "Formatted at 2026-08-06 10:47:54 on dist-test-slave-0b58"
server_key: "eb80e4ccae5bc0aa49ba99f155664881"
server_key_iv: "757c19d060f2c4d57ef699cb3df765a2"
server_key_version: "encryptionkey@0"
I20260806 10:47:54.532670 22802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 10:47:54.535686 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:54.536942 22802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:47:54.537107 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce"
format_stamp: "Formatted at 2026-08-06 10:47:54 on dist-test-slave-0b58"
server_key: "eb80e4ccae5bc0aa49ba99f155664881"
server_key_iv: "757c19d060f2c4d57ef699cb3df765a2"
server_key_version: "encryptionkey@0"
I20260806 10:47:54.537283 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:54.600522 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:54.604482 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:54.604784 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:54.605764 22802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:54.607105 22802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:54.607213 22802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:54.607366 22802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:54.607450 22802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:54.641521 22802 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:34731
I20260806 10:47:54.641561 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:34731 every 8 connection(s)
I20260806 10:47:54.642814 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:54.650668 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22802
I20260806 10:47:54.650879 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:47:54.651777 16468 external_mini_cluster.cc:1428] Setting key c1aacee68471ea806390b3db7f4c62ab
Aug 06 10:47:54 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013274, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:47:54.663600 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:54.645190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:49547 (local address 127.16.21.62:45475)
0806 10:47:54.645653 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:54.645657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:54.646429 (+ 772us) server_negotiation.cc:407] Connection header received
0806 10:47:54.648235 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:54.648240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:54.648361 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:54.648484 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:54.649910 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:54.650704 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:54.653069 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:54.653717 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:54.653848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:54.657919 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:54.657924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:54.657928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:54.657959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:54.660635 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:54.661346 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:54.661353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:54.661359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:54.661480 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:54.661897 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:54.661903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:54.661907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:54.662156 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:54.662331 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:54.663008 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 10:47:54.663238 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":248,"threads_started":1}
I20260806 10:47:54.664579 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:54.645575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:49547)
0806 10:47:54.646266 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:54.646312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:54.647974 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:54.648713 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:54.648721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:54.649204 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:54.649728 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:54.649745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:54.650951 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:54.650959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:54.652920 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:54.652932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:54.653973 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:54.653980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:54.654254 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:54.654964 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:54.654999 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:54.657699 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:54.660777 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:54.660785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:54.660800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:54.661217 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:54.661591 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:54.661596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:54.661599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:54.661787 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:54.662311 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:54.662319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:54.662634 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 10:47:54.662917 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 10:47:54.663425 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":282,"threads_started":1}
I20260806 10:47:54.666075 22931 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45475
I20260806 10:47:54.666407 22931 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:54.667249 22931 heartbeater.cc:511] Master 127.16.21.62:45475 requested a full tablet report, sending...
I20260806 10:47:54.669620 22741 ts_manager.cc:194] Registered new tserver with Master: e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731)
I20260806 10:47:54.671767 22741 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:49547
WARNING: no policy specified for oryx/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab.
Entry for principal oryx/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab.
I20260806 10:47:54.709450 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45475
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:55.045460 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:55.045711 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:55.045804 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:55.052986 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:55.053079 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:55.053195 22938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:55.062486 22938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:55.063555 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:55.064872 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:55.077750 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:55.077865 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:55.078241 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:55.078842 22938 server_base.cc:1034] running on GCE node
I20260806 10:47:55.079452 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:55.080345 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:55.081574 22938 hybrid_clock.cc:648] HybridClock initialized: now 1786013275081528 us; error 50 us; skew 500 ppm
I20260806 10:47:55.081895 22938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:55 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013275, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:55.085137 22938 init.cc:377] Logged in from keytab as oryx/127.16.21.2@KRBTEST.COM (short username oryx)
I20260806 10:47:55.086594 22938 webserver.cc:460] Webserver started at http://127.16.21.2:34643/ using document root <none> and password file <none>
I20260806 10:47:55.087230 22938 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:55.087373 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:55.087666 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:55.090591 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "2f96449697744f509e9262d4e1ce283b"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "7b624a2cc655adfdd4932a15dc745648"
server_key_iv: "c6fd6f7809ecccfe735c65266436b5e0"
server_key_version: "encryptionkey@0"
I20260806 10:47:55.091395 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "2f96449697744f509e9262d4e1ce283b"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "7b624a2cc655adfdd4932a15dc745648"
server_key_iv: "c6fd6f7809ecccfe735c65266436b5e0"
server_key_version: "encryptionkey@0"
I20260806 10:47:55.096306 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260806 10:47:55.099555 22953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:55.100833 22938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:55.100997 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "2f96449697744f509e9262d4e1ce283b"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "7b624a2cc655adfdd4932a15dc745648"
server_key_iv: "c6fd6f7809ecccfe735c65266436b5e0"
server_key_version: "encryptionkey@0"
I20260806 10:47:55.101184 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:55.114888 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:55.118952 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:55.119248 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:55.120210 22938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:55.121487 22938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:55.121585 22938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:55.121729 22938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:55.121811 22938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:55.155689 22938 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:45963
I20260806 10:47:55.155789 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:45963 every 8 connection(s)
I20260806 10:47:55.156980 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:55.160681 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 22938
I20260806 10:47:55.160882 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:47:55.161517 16468 external_mini_cluster.cc:1428] Setting key 51486006ec7f87d7feb9003ff65e7c62
Aug 06 10:47:55 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013275, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:47:55.178614 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:55.159499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:48993 (local address 127.16.21.62:45475)
0806 10:47:55.159669 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:55.159673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:55.161026 (+ 1353us) server_negotiation.cc:407] Connection header received
0806 10:47:55.163427 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:55.163432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:55.163535 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:55.163611 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:55.165128 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.166359 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.167868 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.168610 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.168754 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:55.172038 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:55.172046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:55.172054 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:55.172098 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:55.175604 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:55.176335 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:55.176343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:55.176346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:55.176491 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:55.176952 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:55.176956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:55.176958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:55.177215 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:55.177374 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:55.178031 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 10:47:55.178266 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:47:55.179491 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:55.159983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:48993)
0806 10:47:55.160860 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:55.160910 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:55.163150 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:55.163864 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:55.163872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:55.164439 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:55.164948 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:55.164964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.166514 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.166517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:55.167700 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:55.167714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.168852 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.168857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:55.169066 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:55.169740 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:55.169765 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:55.171802 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:55.175748 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:55.175757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:55.175770 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:55.176180 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:55.176608 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:55.176613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:55.176617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:55.176821 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:55.177358 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:55.177365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:55.177671 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 10:47:55.177934 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 10:47:55.178432 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260806 10:47:55.180941 23066 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45475
I20260806 10:47:55.181257 23066 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:55.181977 23066 heartbeater.cc:511] Master 127.16.21.62:45475 requested a full tablet report, sending...
I20260806 10:47:55.183421 22741 ts_manager.cc:194] Registered new tserver with Master: 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:47:55.184171 22741 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:48993
WARNING: no policy specified for oryx/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab.
Entry for principal oryx/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab.
I20260806 10:47:55.219828 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45475
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:55.555375 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:55.555622 23071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:55.555809 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:55.563076 23071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:55.563198 23071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:55.563387 23071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:55.572788 23071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:55.574003 23071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:55.575431 23071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:55.588825 23077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:55.588778 23080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:55.588860 23078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:55.589687 23071 server_base.cc:1034] running on GCE node
I20260806 10:47:55.590271 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:55.591147 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:55.592381 23071 hybrid_clock.cc:648] HybridClock initialized: now 1786013275592322 us; error 32 us; skew 500 ppm
I20260806 10:47:55.592747 23071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:55 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013275, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:55.601475 23071 init.cc:377] Logged in from keytab as oryx/127.16.21.3@KRBTEST.COM (short username oryx)
I20260806 10:47:55.602877 23071 webserver.cc:460] Webserver started at http://127.16.21.3:45795/ using document root <none> and password file <none>
I20260806 10:47:55.603523 23071 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:55.603658 23071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:55.603971 23071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:55.606948 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "fc046a7def484b08a6b2e2ae73728e04"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "b1038499cb58c8e448dc4b556df05fc0"
server_key_iv: "945d15fddbcc84a17115c23911b0300e"
server_key_version: "encryptionkey@0"
I20260806 10:47:55.607739 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "fc046a7def484b08a6b2e2ae73728e04"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "b1038499cb58c8e448dc4b556df05fc0"
server_key_iv: "945d15fddbcc84a17115c23911b0300e"
server_key_version: "encryptionkey@0"
I20260806 10:47:55.612898 23071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 10:47:55.616173 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:55.617456 23071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:47:55.617616 23071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "fc046a7def484b08a6b2e2ae73728e04"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "b1038499cb58c8e448dc4b556df05fc0"
server_key_iv: "945d15fddbcc84a17115c23911b0300e"
server_key_version: "encryptionkey@0"
I20260806 10:47:55.617838 23071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:55.632769 23071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:55.636433 23071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:55.636721 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:55.637676 23071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:55.638978 23071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:55.639051 23071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:55.639230 23071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:55.639288 23071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:55.673785 23071 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41205
I20260806 10:47:55.673833 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41205 every 8 connection(s)
I20260806 10:47:55.675133 23071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:55.675352 22931 heartbeater.cc:503] Master 127.16.21.62:45475 was elected leader, sending a full tablet report...
I20260806 10:47:55.681941 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 23071
I20260806 10:47:55.682132 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:47:55.682578 16468 external_mini_cluster.cc:1428] Setting key 9b29aeb3e172e2ce62f6617f47da75ea
Aug 06 10:47:55 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013275, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:47:55.694887 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:55.677824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:39887 (local address 127.16.21.62:45475)
0806 10:47:55.678001 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:55.678005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:55.679205 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 10:47:55.681044 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:55.681048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:55.681153 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:55.681268 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:55.682885 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.683775 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.685543 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.685948 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.686080 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:55.689372 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:55.689377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:55.689380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:55.689413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:55.691750 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:55.692525 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:55.692529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:55.692532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:55.692657 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:55.693100 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:55.693105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:55.693107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:55.693518 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:55.693672 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:55.694313 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 10:47:55.694523 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:47:55.695741 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:55.678319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:39887)
0806 10:47:55.679061 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:55.679106 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:55.680781 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:55.681476 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:55.681484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:55.682028 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:55.682692 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:55.682717 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.683945 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.683955 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:55.685393 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:55.685405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:55.686201 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:55.686209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:55.686443 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:55.687044 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:55.687065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:55.689167 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:55.691890 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:55.691898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:55.691911 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:55.692396 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:55.692781 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:55.692786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:55.692789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:55.692970 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:55.693647 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:55.693654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:55.693962 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 10:47:55.694198 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 10:47:55.694667 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":284,"threads_started":1}
I20260806 10:47:55.697255 23201 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45475
I20260806 10:47:55.697589 23201 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:55.698311 23201 heartbeater.cc:511] Master 127.16.21.62:45475 requested a full tablet report, sending...
I20260806 10:47:55.699714 22740 ts_manager.cc:194] Registered new tserver with Master: fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205)
I20260806 10:47:55.700598 22740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:39887
I20260806 10:47:55.709674 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.61.keytab.
Entry for principal oryx/127.16.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.61.keytab.
I20260806 10:47:55.768128 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:39685
--webserver_interface=127.16.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.61.keytab
--principal=oryx/127.16.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45475,127.16.21.61:39685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:56.099989 23206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:56.100334 23206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:56.100504 23206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:56.107985 23206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:56.108119 23206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:56.108235 23206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:56.108361 23206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:56.108588 23206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:56.118395 23206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45475,127.16.21.61:39685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.61.keytab
--principal=oryx/127.16.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:39685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
W20260806 10:47:56.118786 23206 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.21.62:45475,127.16.21.61:39685'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 10:47:56.119779 23206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:56.121240 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:56.132969 23213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:56.133052 23212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:56.133016 23216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:56.133893 23206 server_base.cc:1034] running on GCE node
I20260806 10:47:56.134588 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:56.135816 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:56.137058 23206 hybrid_clock.cc:648] HybridClock initialized: now 1786013276137006 us; error 30 us; skew 500 ppm
I20260806 10:47:56.137385 23206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:56.140575 23206 init.cc:377] Logged in from keytab as oryx/127.16.21.61@KRBTEST.COM (short username oryx)
I20260806 10:47:56.142072 23206 webserver.cc:460] Webserver started at http://127.16.21.61:35957/ using document root <none> and password file <none>
I20260806 10:47:56.142729 23206 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:56.142855 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:56.143147 23206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:56.146087 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/instance:
uuid: "4525dcf585fa4701b6c7623f391e956f"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "0b06c20d0021de44d235fe6a3a97d45f"
server_key_iv: "d6b1b7d4db2ef362f4428c3f58b6deff"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.146857 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal/instance:
uuid: "4525dcf585fa4701b6c7623f391e956f"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "0b06c20d0021de44d235fe6a3a97d45f"
server_key_iv: "d6b1b7d4db2ef362f4428c3f58b6deff"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.151778 23206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:47:56.155344 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:56.156713 23206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:56.156885 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
uuid: "4525dcf585fa4701b6c7623f391e956f"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "0b06c20d0021de44d235fe6a3a97d45f"
server_key_iv: "d6b1b7d4db2ef362f4428c3f58b6deff"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.157089 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:56.181969 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:56.186321 23206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:56.186651 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:56.186767 23066 heartbeater.cc:503] Master 127.16.21.62:45475 was elected leader, sending a full tablet report...
I20260806 10:47:56.205863 23206 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.61:39685
I20260806 10:47:56.205957 23275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.61:39685 every 8 connection(s)
I20260806 10:47:56.207522 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
I20260806 10:47:56.211050 23276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:56.211109 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 23206
I20260806 10:47:56.220719 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
I20260806 10:47:56.237419 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.213287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.16.21.61:39685)
0806 10:47:56.214130 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.214141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.214184 (+ 43us) server_negotiation.cc:407] Connection header received
0806 10:47:56.215635 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.215653 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.216011 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.216405 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.218214 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.219496 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.221466 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.222307 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.222496 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.228728 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.228740 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.228758 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.228805 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.233070 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.233841 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.233849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.233856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.234013 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.234471 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.234474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.234477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.235022 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.235274 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.235674 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.236044 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":645,"thread_start_us":400,"threads_started":1}
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.61@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:47:56.245579 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.223884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.16.21.62:45475)
0806 10:47:56.230594 (+ 6710us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.230600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.230623 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:47:56.230688 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.230696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.230802 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.230917 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.233128 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.234168 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.235947 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.236616 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.236752 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.239959 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.239967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.239977 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.240008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.242748 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.243484 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.243490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.243493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.243622 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.244019 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.244025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.244029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.244419 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.244574 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.245012 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.245219 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6578,"thread_start_us":6429,"threads_started":1}
I20260806 10:47:56.245621 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.223757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:38864)
0806 10:47:56.228727 (+ 4970us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.228780 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.228967 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.231020 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.231027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.232530 (+ 1503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.232936 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.232950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.234317 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.234322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.235785 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.235802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.236808 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.236813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.236954 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.237553 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.237591 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.239721 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.242893 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.242904 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.242910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.243352 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.243739 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.243744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.243747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.243908 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.244546 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.244553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.244722 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.244909 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.245208 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4649,"thread_start_us":5830,"threads_started":1}
I20260806 10:47:56.247293 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.61@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
I20260806 10:47:56.258931 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.247845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:45468)
0806 10:47:56.248048 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.248068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.248202 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.248566 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.248570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.248924 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.249061 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.249069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.249999 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.250004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.250922 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.250933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.251633 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.251639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.251799 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.252291 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.252318 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.254226 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.256964 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.256968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.256970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.257319 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.257632 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.257635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.257637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.257765 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.258287 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.258290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.258406 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.258512 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.258700 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 10:47:56.259151 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.247900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.16.21.61:39685)
0806 10:47:56.248049 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.248055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.248144 (+ 89us) server_negotiation.cc:407] Connection header received
0806 10:47:56.248385 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.248390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.248453 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.248548 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.249191 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.249860 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.251062 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.251482 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.251610 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.254400 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.254404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.254407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.254434 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.256847 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.257424 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.257427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.257429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.257540 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.257890 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.257894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.257896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.258191 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.258300 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.258597 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.258829 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":56}
I20260806 10:47:56.263706 23276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Bootstrap starting.
I20260806 10:47:56.266583 23276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:56.267534 23276 log.cc:824] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:56.270078 23276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: No bootstrap required, opened a new log
I20260806 10:47:56.276795 23276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } }
I20260806 10:47:56.277062 23276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:56.277136 23276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4525dcf585fa4701b6c7623f391e956f, State: Initialized, Role: FOLLOWER
I20260806 10:47:56.277683 23276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } }
I20260806 10:47:56.278435 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } } }
I20260806 10:47:56.278712 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:47:56.279371 23276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:47:56.287647 23294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:47:56.287782 23294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:47:56.290406 23206 master_runner.cc:428] Detected that this master 4525dcf585fa4701b6c7623f391e956f is joining an existing cluster
I20260806 10:47:56.290522 23206 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.21.61:39685
I20260806 10:47:56.318954 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.298253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.16.21.62:45475)
0806 10:47:56.298462 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.298471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.299156 (+ 685us) server_negotiation.cc:407] Connection header received
0806 10:47:56.299494 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.299503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.299612 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.299702 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.301607 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.303068 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.306248 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.307205 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.307341 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.310581 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.310590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.310597 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.310647 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.315557 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.316561 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.316568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.316574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.316709 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.317169 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.317174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.317176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.317530 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.317706 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.318329 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.318572 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:47:56.320475 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.298348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:38870)
0806 10:47:56.299057 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.299083 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.299296 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.299994 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.300002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.300908 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.301395 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.301412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.303290 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.303298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.306007 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.306028 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.307738 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.307748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.308004 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.308811 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.308844 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.310315 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.315777 (+ 5462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.315785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.315793 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.316418 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.316861 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.316867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.316870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.317042 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.317809 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.317824 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.318018 (+ 194us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.319493 (+ 1475us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.319861 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":401,"threads_started":1}
WARNING: no policy specified for oryx/127.16.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.60@KRBTEST.COM" created.
I20260806 10:47:56.335726 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.299294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:45484)
0806 10:47:56.316751 (+ 17457us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.316776 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.316909 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.317355 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.317360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.318059 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.318407 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.318419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.319907 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.319912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.321412 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.321424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.323041 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.323048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.323170 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.323734 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.323751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.324597 (+ 846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.329136 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.329140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.329144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.329672 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.330113 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.330120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.330126 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.330307 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.334378 (+ 4071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.334384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.334563 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.334706 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.335061 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17304,"thread_start_us":17113,"threads_started":1}
I20260806 10:47:56.337007 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.299446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.16.21.61:39685)
0806 10:47:56.299616 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.299621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.316876 (+ 17255us) server_negotiation.cc:407] Connection header received
0806 10:47:56.317024 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.317029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.317102 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.317187 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.318552 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.319653 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.321579 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.322320 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.322449 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.324816 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.324822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.324827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.324862 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.328991 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.329814 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.329820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.329825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.329975 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.333651 (+ 3676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.333658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.333663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.334144 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.334321 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.336282 (+ 1961us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.336563 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.62@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
I20260806 10:47:56.356055 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.329692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.16.21.61:39685)
0806 10:47:56.337592 (+ 7900us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.337599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.341855 (+ 4256us) server_negotiation.cc:407] Connection header received
0806 10:47:56.342281 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.342286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.342400 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.342463 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.343885 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.344617 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.345935 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.346545 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.346671 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.349891 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.349895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.349898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.349976 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.353005 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.353710 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.353716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.353728 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.353961 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.354412 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.354417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.354421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.354761 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.354918 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.355477 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.355681 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7772,"thread_start_us":11368,"threads_started":1}
I20260806 10:47:56.356110 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.329353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:45496)
0806 10:47:56.341740 (+ 12387us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.341781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.342016 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.342734 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.342741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.343506 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.343715 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.343726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.344745 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.344753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.345772 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.345785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.346747 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.346753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.346920 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.347560 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.347582 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.349701 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.353173 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.353182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.353187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.353570 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.354079 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.354084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.354093 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.354299 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.354981 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.354987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.355153 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.355376 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.355697 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12140,"thread_start_us":13272,"threads_started":1}
I20260806 10:47:56.358074 22740 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.21.61:39685: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4525dcf585fa4701b6c7623f391e956f"
member_type: NON_VOTER
last_known_addr {
host: "127.16.21.61"
port: 39685
}
attrs {
promote: true
}
}
dest_uuid: "ece5d271795e4851b5e02b6f915b0710"
cas_config_opid_index: -1
I20260806 10:47:56.359169 22740 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } }
I20260806 10:47:56.362664 23308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4525dcf585fa4701b6c7623f391e956f (127.16.21.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.363505 23251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.60.keytab.
Entry for principal oryx/127.16.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.60.keytab.
I20260806 10:47:56.366489 23308 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.21.61:39685
I20260806 10:47:56.367444 23251 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Refusing update from remote peer ece5d271795e4851b5e02b6f915b0710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 10:47:56.367385 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.367707 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:56.368046 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:41973
--webserver_interface=127.16.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.60.keytab
--principal=oryx/127.16.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:47:56.368432 23308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:56.372749 23206 master.cc:537] Master@127.16.21.61:39685 shutting down...
I20260806 10:47:56.393668 23251 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4525dcf585fa4701b6c7623f391e956f (127.16.21.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.400875 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.401580 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:56.401922 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.402612 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:56.402304 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece5d271795e4851b5e02b6f915b0710. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.402920 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:56.422713 23206 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 LEARNER]: Raft consensus shutting down.
I20260806 10:47:56.423213 23206 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 LEARNER]: Raft consensus is shut down!
I20260806 10:47:56.423417 23206 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: stopping tablet replica
I20260806 10:47:56.441213 23206 master.cc:552] Master@127.16.21.61:39685 shutdown complete.
I20260806 10:47:56.441461 23206 master_runner.cc:305] Clearing existing system tablet
I20260806 10:47:56.441947 23206 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:56.442054 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:56.444464 23206 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 10:47:56.445979 23318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:56.446328 23206 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:56.446633 23206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 10:47:56.446735 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
uuid: "4525dcf585fa4701b6c7623f391e956f"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "0b06c20d0021de44d235fe6a3a97d45f"
server_key_iv: "d6b1b7d4db2ef362f4428c3f58b6deff"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.449982 23206 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:56.455111 23206 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 10:47:56.455307 23206 log.cc:1196] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 10:47:56.455801 23206 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Deleting consensus metadata
I20260806 10:47:56.456050 23206 master_runner.cc:315] Copying system tablet from 127.16.21.62:45475
I20260806 10:47:56.464670 23206 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.21.62:45475
I20260806 10:47:56.480158 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.465427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.16.21.62:45475)
0806 10:47:56.465635 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.465644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.468493 (+ 2849us) server_negotiation.cc:407] Connection header received
0806 10:47:56.468732 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.468738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.468821 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.468932 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.469846 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.470606 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.471868 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.472342 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.472471 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.474542 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.474546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.474550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.474630 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.477361 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.478036 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.478043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.478046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.478206 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.478641 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.478647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.478650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.478954 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.479124 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.479556 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.479775 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:47:56.480235 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.465423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:38878)
0806 10:47:56.468374 (+ 2951us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.468391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.468562 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.468959 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.468964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.469436 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.469663 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.469673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.470780 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.470785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.471719 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.471731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.472513 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.472519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.472721 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.473411 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.473428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.474315 (+ 887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.477497 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.477503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.477507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.477884 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.478333 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.478338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.478342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.478513 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.479097 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.479102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.479255 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.479407 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.479726 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2778,"thread_start_us":2622,"threads_started":1}
I20260806 10:47:56.481420 22761 tablet_copy_service.cc:140] P ece5d271795e4851b5e02b6f915b0710: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4525dcf585fa4701b6c7623f391e956f ({username='oryx', principal='oryx/127.16.21.61@KRBTEST.COM'} at 127.0.0.1:38878)
I20260806 10:47:56.481679 22761 tablet_copy_service.cc:161] P ece5d271795e4851b5e02b6f915b0710: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4525dcf585fa4701b6c7623f391e956f at {username='oryx', principal='oryx/127.16.21.61@KRBTEST.COM'} at 127.0.0.1:38878: session id = 4525dcf585fa4701b6c7623f391e956f-00000000000000000000000000000000
I20260806 10:47:56.484752 22761 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 10:47:56.486497 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 10:47:56.491286 23206 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: tablet copy: Starting download of 0 data blocks...
I20260806 10:47:56.491585 23206 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: tablet copy: Starting download of 1 WAL segments...
I20260806 10:47:56.493896 23206 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 10:47:56.497009 22761 tablet_copy_service.cc:342] P ece5d271795e4851b5e02b6f915b0710: Request end of tablet copy session 4525dcf585fa4701b6c7623f391e956f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.21.61@KRBTEST.COM'} at 127.0.0.1:38878
I20260806 10:47:56.497181 22761 tablet_copy_service.cc:433] P ece5d271795e4851b5e02b6f915b0710: ending tablet copy session 4525dcf585fa4701b6c7623f391e956f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4525dcf585fa4701b6c7623f391e956f
I20260806 10:47:56.507296 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:56.510054 23326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:56.515214 23327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:56.523218 23206 server_base.cc:1034] running on GCE node
W20260806 10:47:56.523339 23329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:56.523932 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:56.524650 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:56.525911 23206 hybrid_clock.cc:648] HybridClock initialized: now 1786013276525854 us; error 32 us; skew 500 ppm
I20260806 10:47:56.526136 23206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:56.528064 23206 init.cc:377] Logged in from keytab as oryx/127.16.21.61@KRBTEST.COM (short username oryx)
I20260806 10:47:56.529676 23206 webserver.cc:460] Webserver started at http://127.16.21.61:37767/ using document root <none> and password file <none>
I20260806 10:47:56.530033 23206 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:56.530136 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:56.532467 23206 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:47:56.534072 23336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:56.534492 23206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 10:47:56.534617 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
uuid: "4525dcf585fa4701b6c7623f391e956f"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "0b06c20d0021de44d235fe6a3a97d45f"
server_key_iv: "d6b1b7d4db2ef362f4428c3f58b6deff"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.534785 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:56.544476 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:56.548043 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:56.577288 23206 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.61:39685
I20260806 10:47:56.577293 23387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.61:39685 every 8 connection(s)
I20260806 10:47:56.578245 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
I20260806 10:47:56.586328 23388 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:47:56.590119 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Bootstrap starting.
I20260806 10:47:56.601094 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:47:56.601550 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: Bootstrap complete.
I20260806 10:47:56.602391 23388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } }
I20260806 10:47:56.602617 23388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4525dcf585fa4701b6c7623f391e956f, State: Initialized, Role: LEARNER
I20260806 10:47:56.602881 23388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } }
I20260806 10:47:56.603762 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.604051 23388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:56.604133 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:56.612970 23402 catalog_manager.cc:1259] Loaded cluster ID: 2a75c74a8c5b42cdaa63240d0a22fb29
I20260806 10:47:56.613081 23402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: loading cluster ID for follower catalog manager: success
I20260806 10:47:56.615446 23402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: acquiring CA information for follower catalog manager: success
I20260806 10:47:56.617352 23402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:47:56.702944 23201 heartbeater.cc:503] Master 127.16.21.62:45475 was elected leader, sending a full tablet report...
I20260806 10:47:56.756534 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.756846 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:56.766717 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.757320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:45508)
0806 10:47:56.757755 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.757792 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.757933 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.759369 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.759377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.759415 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:56.759652 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.759668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.761820 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.761826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.763950 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.763969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.765848 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.765855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.766083 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.766148 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.766275 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.766421 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260806 10:47:56.766840 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.757843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.16.21.61:39685)
0806 10:47:56.758886 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.758893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.758917 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:47:56.759005 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.759013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.759153 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.759291 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:56.759842 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.761650 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.764153 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.765658 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.765809 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.766070 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.766318 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.766468 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":648,"threads_started":1}
I20260806 10:47:56.768245 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece5d271795e4851b5e02b6f915b0710. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: NON_VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: true } } }
I20260806 10:47:56.768569 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:56.769270 23403 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: attempting to promote NON_VOTER 4525dcf585fa4701b6c7623f391e956f to VOTER
I20260806 10:47:56.770256 23403 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } }
I20260806 10:47:56.772346 23363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 LEARNER]: Refusing update from remote peer ece5d271795e4851b5e02b6f915b0710: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 10:47:56.772970 23308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 10:47:56.776361 23403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4525dcf585fa4701b6c7623f391e956f (127.16.21.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } }
I20260806 10:47:56.780459 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } }
I20260806 10:47:56.780679 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:56.780550 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } }
I20260806 10:47:56.780802 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:56.780069 23363 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4525dcf585fa4701b6c7623f391e956f (127.16.21.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } }
I20260806 10:47:56.784869 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } }
I20260806 10:47:56.785089 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: FOLLOWER
W20260806 10:47:56.793892 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:56.794147 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:56.794346 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:56.801805 23311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:47:56.801903 23311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:56.802001 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:56.802099 23311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:47:56.802163 23311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:47:56.812098 23311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.60.keytab
--principal=oryx/127.16.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:41973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:56.813391 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:56.814649 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:56.826213 23410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:56.826388 23411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:56.826378 23413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:56.828383 23311 server_base.cc:1034] running on GCE node
I20260806 10:47:56.829326 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:56.831116 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:56.832367 23311 hybrid_clock.cc:648] HybridClock initialized: now 1786013276832316 us; error 37 us; skew 500 ppm
I20260806 10:47:56.832780 23311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:56.836634 23311 init.cc:377] Logged in from keytab as oryx/127.16.21.60@KRBTEST.COM (short username oryx)
I20260806 10:47:56.840968 23311 webserver.cc:460] Webserver started at http://127.16.21.60:39989/ using document root <none> and password file <none>
I20260806 10:47:56.841614 23311 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:56.841763 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:56.842103 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:47:56.845046 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/instance:
uuid: "7108e67ae9d0454b84f38a9095e42bb8"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "22995b60507f37191b671f29a9bbee3f"
server_key_iv: "28cca065e097ac682bce116236cad615"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.845818 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal/instance:
uuid: "7108e67ae9d0454b84f38a9095e42bb8"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "22995b60507f37191b671f29a9bbee3f"
server_key_iv: "28cca065e097ac682bce116236cad615"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.850929 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:47:56.854393 23420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:56.855764 23311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:56.855942 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
uuid: "7108e67ae9d0454b84f38a9095e42bb8"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "22995b60507f37191b671f29a9bbee3f"
server_key_iv: "28cca065e097ac682bce116236cad615"
server_key_version: "encryptionkey@0"
I20260806 10:47:56.856132 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:56.882938 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:56.887050 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:56.887331 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:56.906054 23311 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.60:41973
I20260806 10:47:56.906137 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.60:41973 every 8 connection(s)
I20260806 10:47:56.907454 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
I20260806 10:47:56.911979 23473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:56.917310 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 23311
I20260806 10:47:56.921351 23473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.60@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.60@KRBTEST.COM
I20260806 10:47:56.941227 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.925652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.16.21.62:45475)
0806 10:47:56.925848 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.925855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.926512 (+ 657us) server_negotiation.cc:407] Connection header received
0806 10:47:56.926795 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.926801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.926880 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.926992 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.928603 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.929543 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.931285 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.931750 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.931892 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.935166 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.935170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.935173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.935205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.937969 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.938855 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.938863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.938867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.939016 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.939424 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.939429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.939433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.939684 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.939838 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.940609 (+ 771us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.940828 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 10:47:56.942216 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.919583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.16.21.60:41973)
0806 10:47:56.920764 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.920802 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.920850 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:47:56.922682 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.922710 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.923164 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.923589 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.925528 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.927551 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.929105 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.930228 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.930436 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.935970 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.935978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.935988 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.936015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.938900 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.939576 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.939581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.939583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.939700 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.940116 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.940119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.940121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.940537 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.940882 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.941654 (+ 772us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.941815 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"spinlock_wait_cycles":17280,"thread_start_us":739,"threads_started":1}
I20260806 10:47:56.942257 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.925249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:38892)
0806 10:47:56.926365 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.926411 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.926640 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.927067 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.927075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.928034 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.928434 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.928449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.929684 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.929689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.931094 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.931106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.931987 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.931994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.932151 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.932774 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.932794 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.934926 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.938109 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.938120 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.938130 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.938723 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.939139 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.939144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.939148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.939314 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.939821 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.939827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.940191 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.940523 (+ 332us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.941070 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":643,"threads_started":1}
I20260806 10:47:56.943665 23473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.60@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
I20260806 10:47:56.955230 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.944212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:45518)
0806 10:47:56.944426 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.944443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.944530 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.945016 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.945019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.945296 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.945410 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.945416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.946364 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.946367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.947233 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.947242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.947875 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.947880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.948021 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.948524 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.948539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.950321 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.953194 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.953197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.953200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.953546 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.953923 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.953926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.953928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.954047 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.954541 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.954544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.954645 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.954804 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.955008 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 10:47:56.955360 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.944428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.16.21.61:39685)
0806 10:47:56.944610 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.944614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.944629 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:47:56.944675 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.944678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.944733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.944846 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.945541 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.946250 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.947347 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.947733 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.947826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.950499 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.950503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.950506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.950535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.953091 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.953650 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.953674 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.953679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.953829 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.954137 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.954153 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.954174 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.954445 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.954553 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.954895 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.955098 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:47:56.956164 23473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:47:56 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013276, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.60@KRBTEST.COM for oryx/127.16.21.60@KRBTEST.COM
I20260806 10:47:56.967712 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.956782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.0.0.1:36346)
0806 10:47:56.956982 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:56.956997 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:56.957073 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:56.957310 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:56.957313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:56.957613 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:56.957745 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.957751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.958765 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.958768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.959738 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:56.959763 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.960464 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.960468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:56.960629 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.961056 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:56.961070 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:56.962811 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:56.965441 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.965448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.965452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.965772 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.966117 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:56.966123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:56.966126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:56.966305 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:56.966768 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:56.966780 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:56.966904 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 10:47:56.967033 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 10:47:56.967285 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260806 10:47:56.967730 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.956841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.16.21.60:41973)
0806 10:47:56.956999 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.957003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.957035 (+ 32us) server_negotiation.cc:407] Connection header received
0806 10:47:56.957154 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.957157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.957204 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.957284 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:56.957865 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.958657 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.959859 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:56.960347 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:56.960469 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:56.962984 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:56.962990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:56.962993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:56.963019 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:56.965317 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.965893 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.965899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.965907 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.966016 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:56.966407 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:56.966414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:56.966417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:56.966647 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:56.966788 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:56.967114 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 10:47:56.967434 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":50}
I20260806 10:47:56.971953 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Bootstrap starting.
I20260806 10:47:56.975193 23473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:56.976109 23473 log.cc:824] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:56.978451 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: No bootstrap required, opened a new log
I20260806 10:47:56.985180 23473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } }
I20260806 10:47:56.985461 23473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:56.985543 23473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7108e67ae9d0454b84f38a9095e42bb8, State: Initialized, Role: FOLLOWER
I20260806 10:47:56.986001 23473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } }
I20260806 10:47:56.986734 23479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } } }
I20260806 10:47:56.987059 23479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:47:56.987663 23473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:47:56.996119 23490 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:47:56.996321 23490 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:47:56.996927 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22802
I20260806 10:47:57.000598 23311 master_runner.cc:428] Detected that this master 7108e67ae9d0454b84f38a9095e42bb8 is joining an existing cluster
I20260806 10:47:57.000765 23311 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.21.60:41973
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.1@KRBTEST.COM
I20260806 10:47:57.018775 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:56.998083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.16.21.1:34731)
0806 10:47:56.998733 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:56.998738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:56.998761 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:47:56.998840 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:56.998848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:56.999072 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:56.999237 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.000753 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.001899 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.003615 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.004410 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.004574 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.011387 (+ 6813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.011405 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.011419 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.011458 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.015271 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.016140 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.016149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.016153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.016325 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.016782 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.016787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.016790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.017199 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.017461 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.018137 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.018382 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":335,"threads_started":1}
I20260806 10:47:57.021023 22926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:57.024727 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.006747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:45532)
0806 10:47:57.008618 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.008638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.009059 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.009571 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.009581 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.010242 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.010613 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.010635 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.012091 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.012097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.013857 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.013873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.015109 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.015116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.015325 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.015947 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.015969 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.017123 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.021318 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.021323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.021326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.021789 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.022449 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.022457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.022463 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.022699 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.023460 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.023468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.023741 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.023946 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.024223 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"mutex_wait_us":1,"thread_start_us":1506,"threads_started":1}
I20260806 10:47:57.024752 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.006421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.16.21.61:39685)
0806 10:47:57.006616 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.006623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.008719 (+ 2096us) server_negotiation.cc:407] Connection header received
0806 10:47:57.009202 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.009208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.009293 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.009360 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.010823 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.011883 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.014024 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.014721 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.014831 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.017336 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.017342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.017347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.017392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.021135 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.021999 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.022008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.022027 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.022249 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.022873 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.022882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.022888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.023324 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.023486 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.024100 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.024367 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:47:57.040182 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.008814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.16.21.60:41973)
0806 10:47:57.008979 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.008985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.025382 (+ 16397us) server_negotiation.cc:407] Connection header received
0806 10:47:57.025497 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.025504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.025583 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.025686 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.026542 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.027685 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.029878 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.030522 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.030643 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.033242 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.033248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.033252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.033284 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.037446 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.038168 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.038173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.038176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.038285 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.038625 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.038630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.038633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.039025 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.039166 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.039583 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.039784 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:47:57.041638 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.008735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.0.0.1:36358)
0806 10:47:57.025276 (+ 16541us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.025296 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.025393 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.025724 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.025729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.026205 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.026391 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.026401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.027885 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.027891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.029696 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.029715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.030691 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.030699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.031002 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.031626 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.031659 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.033037 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.037575 (+ 4538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.037579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.037583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.038059 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.038391 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.038395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.038399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.038511 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.039220 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.039224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.039368 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.040702 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.041184 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16450,"threads_started":1}
I20260806 10:47:57.049196 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.005203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.16.21.62:45475)
0806 10:47:57.005395 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.005401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.006239 (+ 838us) server_negotiation.cc:407] Connection header received
0806 10:47:57.006503 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.006510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.006589 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.006657 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.009058 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.010235 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.012450 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.013175 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.013295 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.041422 (+ 28127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.041431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.041438 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.041498 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.045965 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.046854 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.046859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.046863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.047026 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.047483 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.047488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.047492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.047821 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.047973 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.048615 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.048838 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:47:57.050310 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.005286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:38894)
0806 10:47:57.006091 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.006110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.006355 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.006864 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.006869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.007592 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.008900 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.008914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.010425 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.010430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.012225 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.012242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.031127 (+ 18885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.031134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.031351 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.033209 (+ 1858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.033230 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.040786 (+ 7556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.046209 (+ 5423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.046217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.046223 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.046714 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.047162 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.047166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.047169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.047363 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.048049 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.048054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.048284 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.049537 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.049900 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":454,"threads_started":1}
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.62@KRBTEST.COM for oryx/127.16.21.60@KRBTEST.COM
I20260806 10:47:57.074292 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.058020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.0.0.1:36370)
0806 10:47:57.058403 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.058423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.058533 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.059317 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.059323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.059837 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.060049 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.060066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.061725 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.061731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.063199 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.063218 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.064244 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.064250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.064448 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.065017 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.065039 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.067695 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.071626 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.071631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.071635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.072094 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.072519 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.072524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.072528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.072669 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.073280 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.073285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.073465 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.073610 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.073852 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260806 10:47:57.074360 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.058517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.16.21.60:41973)
0806 10:47:57.058688 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.058693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.058711 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:47:57.058802 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.058807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.058922 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.059044 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.060298 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.061570 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.063374 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.064079 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.064189 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.067885 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.067892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.067896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.067932 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.071474 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.072234 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.072241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.072244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.072403 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.072790 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.072795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.072799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.073128 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.073260 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.073718 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.074005 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:47:57.076496 22740 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.21.60:41973: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8"
member_type: NON_VOTER
last_known_addr {
host: "127.16.21.60"
port: 41973
}
attrs {
promote: true
}
}
dest_uuid: "ece5d271795e4851b5e02b6f915b0710"
cas_config_opid_index: 6
I20260806 10:47:57.077306 22740 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } }
I20260806 10:47:57.079721 23363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Refusing update from remote peer ece5d271795e4851b5e02b6f915b0710: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 10:47:57.079895 23447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:57.080477 23308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 10:47:57.083160 23308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:57.083635 23447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Refusing update from remote peer ece5d271795e4851b5e02b6f915b0710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 10:47:57.086757 23308 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.21.60:41973
I20260806 10:47:57.087152 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:57.087214 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:57.087426 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:57.087497 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:57.088002 23311 master.cc:537] Master@127.16.21.60:41973 shutting down...
I20260806 10:47:57.087977 23403 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 10:47:57.090808 22728 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 10:47:57.090925 23363 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
W20260806 10:47:57.099572 22728 consensus_peers.cc:487] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 -> Peer 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60:41973): Couldn't send request to peer 7108e67ae9d0454b84f38a9095e42bb8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 10:47:57.101540 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:57.101853 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:47:57.124032 23311 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:57.124514 23311 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:57.124764 23311 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: stopping tablet replica
I20260806 10:47:57.131892 23311 master.cc:552] Master@127.16.21.60:41973 shutdown complete.
I20260806 10:47:57.132006 23311 master_runner.cc:305] Clearing existing system tablet
I20260806 10:47:57.132438 23311 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:57.132614 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:57.136001 23311 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 10:47:57.137794 23504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:57.138177 23311 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:57.138597 23311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:47:57.138764 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
uuid: "7108e67ae9d0454b84f38a9095e42bb8"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "22995b60507f37191b671f29a9bbee3f"
server_key_iv: "28cca065e097ac682bce116236cad615"
server_key_version: "encryptionkey@0"
I20260806 10:47:57.143599 23311 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:57.150631 23311 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 10:47:57.150939 23311 log.cc:1196] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 10:47:57.151459 23311 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Deleting consensus metadata
I20260806 10:47:57.151834 23311 master_runner.cc:315] Copying system tablet from 127.16.21.62:45475
I20260806 10:47:57.154848 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22802
I20260806 10:47:57.169395 23311 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.21.62:45475
I20260806 10:47:57.178296 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:34731
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:45475
--webserver_port=37719
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:47:57.185774 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.170306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.16.21.62:45475)
0806 10:47:57.170542 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.170548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.172816 (+ 2268us) server_negotiation.cc:407] Connection header received
0806 10:47:57.173003 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.173012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.173108 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.173223 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.174693 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.175895 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.177800 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.178228 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.178304 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.180380 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.180388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.180396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.180442 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.183094 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.183744 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.183749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.183751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.183887 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.184340 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.184345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.184348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.184621 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.184730 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.185315 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.185506 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260806 10:47:57.186585 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.170272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:38902)
0806 10:47:57.172687 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.172707 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.172869 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.173426 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.173434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.174124 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.174485 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.174501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.176067 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.176075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.177602 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.177615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.178519 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.178523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.178649 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.179357 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.179374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.180149 (+ 775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.183235 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.183241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.183245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.183602 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.183999 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.184004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.184008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.184168 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.184793 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.184804 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.184982 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.185982 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.186295 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"thread_start_us":2126,"threads_started":1}
I20260806 10:47:57.187150 22761 tablet_copy_service.cc:140] P ece5d271795e4851b5e02b6f915b0710: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7108e67ae9d0454b84f38a9095e42bb8 ({username='oryx', principal='oryx/127.16.21.60@KRBTEST.COM'} at 127.0.0.1:38902)
I20260806 10:47:57.187361 22761 tablet_copy_service.cc:161] P ece5d271795e4851b5e02b6f915b0710: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7108e67ae9d0454b84f38a9095e42bb8 at {username='oryx', principal='oryx/127.16.21.60@KRBTEST.COM'} at 127.0.0.1:38902: session id = 7108e67ae9d0454b84f38a9095e42bb8-00000000000000000000000000000000
I20260806 10:47:57.190996 22761 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 10:47:57.192870 23311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 10:47:57.199651 23311 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: tablet copy: Starting download of 0 data blocks...
I20260806 10:47:57.199937 23311 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: tablet copy: Starting download of 1 WAL segments...
I20260806 10:47:57.202418 23311 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 10:47:57.206634 22761 tablet_copy_service.cc:342] P ece5d271795e4851b5e02b6f915b0710: Request end of tablet copy session 7108e67ae9d0454b84f38a9095e42bb8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.21.60@KRBTEST.COM'} at 127.0.0.1:38902
I20260806 10:47:57.206792 22761 tablet_copy_service.cc:433] P ece5d271795e4851b5e02b6f915b0710: ending tablet copy session 7108e67ae9d0454b84f38a9095e42bb8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7108e67ae9d0454b84f38a9095e42bb8
I20260806 10:47:57.218503 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:57.221035 23512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:57.222290 23513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:57.232503 23311 server_base.cc:1034] running on GCE node
W20260806 10:47:57.233359 23515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:57.233794 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:57.234414 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:57.235659 23311 hybrid_clock.cc:648] HybridClock initialized: now 1786013277235617 us; error 27 us; skew 500 ppm
I20260806 10:47:57.235879 23311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013277, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:57.237982 23311 init.cc:377] Logged in from keytab as oryx/127.16.21.60@KRBTEST.COM (short username oryx)
I20260806 10:47:57.241129 23311 webserver.cc:460] Webserver started at http://127.16.21.60:36663/ using document root <none> and password file <none>
I20260806 10:47:57.241472 23311 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:57.241576 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:57.244580 23311 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 10:47:57.246827 23522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:57.247650 23311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 10:47:57.247828 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
uuid: "7108e67ae9d0454b84f38a9095e42bb8"
format_stamp: "Formatted at 2026-08-06 10:47:56 on dist-test-slave-0b58"
server_key: "22995b60507f37191b671f29a9bbee3f"
server_key_iv: "28cca065e097ac682bce116236cad615"
server_key_version: "encryptionkey@0"
I20260806 10:47:57.248095 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:57.277518 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:57.281553 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:57.309163 23311 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.60:41973
I20260806 10:47:57.309219 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.60:41973 every 8 connection(s)
I20260806 10:47:57.310367 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
I20260806 10:47:57.315786 23574 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:47:57.319826 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Bootstrap starting.
I20260806 10:47:57.339040 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:47:57.339525 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Bootstrap complete.
I20260806 10:47:57.340615 23574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } }
I20260806 10:47:57.340965 23574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7108e67ae9d0454b84f38a9095e42bb8, State: Initialized, Role: LEARNER
I20260806 10:47:57.341297 23574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } }
I20260806 10:47:57.342406 23574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:47:57.342129 23577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:57.342657 23577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:57.350572 23588 catalog_manager.cc:1259] Loaded cluster ID: 2a75c74a8c5b42cdaa63240d0a22fb29
I20260806 10:47:57.350696 23588 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: loading cluster ID for follower catalog manager: success
I20260806 10:47:57.354228 23588 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: acquiring CA information for follower catalog manager: success
I20260806 10:47:57.356611 23588 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 10:47:57.555784 23511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:57.556048 23511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:57.556228 23511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:57.563936 23511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:57.564049 23511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:57.564231 23511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:47:57.574415 23511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:34731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=37719
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:57.575867 23511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:57.577622 23511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:57.592422 23595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:57.592375 23596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:57.593494 23511 server_base.cc:1034] running on GCE node
W20260806 10:47:57.593773 23598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:57.594614 23511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:57.595463 23511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:57.596674 23511 hybrid_clock.cc:648] HybridClock initialized: now 1786013277596624 us; error 37 us; skew 500 ppm
I20260806 10:47:57.597002 23511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013277, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:57.600103 23511 init.cc:377] Logged in from keytab as oryx/127.16.21.1@KRBTEST.COM (short username oryx)
I20260806 10:47:57.601486 23511 webserver.cc:460] Webserver started at http://127.16.21.1:37719/ using document root <none> and password file <none>
I20260806 10:47:57.602100 23511 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:57.602252 23511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:57.607856 23511 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 10:47:57.610558 23605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:57.612010 23511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:47:57.612200 23511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce"
format_stamp: "Formatted at 2026-08-06 10:47:54 on dist-test-slave-0b58"
server_key: "eb80e4ccae5bc0aa49ba99f155664881"
server_key_iv: "757c19d060f2c4d57ef699cb3df765a2"
server_key_version: "encryptionkey@0"
I20260806 10:47:57.613288 23511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:57.629539 23511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:57.633754 23511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:57.634047 23511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:57.635092 23511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:57.636535 23511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:57.636659 23511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:57.636788 23511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:57.636879 23511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:57.671720 23511 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:34731
I20260806 10:47:57.671815 23717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:34731 every 8 connection(s)
I20260806 10:47:57.673336 23511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:47:57.682794 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 23511
I20260806 10:47:57.683168 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22938
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013277, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013277, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013277, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.60@KRBTEST.COM
I20260806 10:47:57.704797 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.677384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:34847 (local address 127.16.21.60:41973)
0806 10:47:57.677963 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.677969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.678157 (+ 188us) server_negotiation.cc:407] Connection header received
0806 10:47:57.680945 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.680952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.681047 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.681129 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.683386 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.684881 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.687175 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.687804 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.687912 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.696138 (+ 8226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.696144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.696149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.696190 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.700832 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.701758 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.701765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.701770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.701941 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.702483 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.702489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.702494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.702849 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.703002 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.704161 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.704399 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":267,"threads_started":1}
I20260806 10:47:57.706194 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.678644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:59519 (local address 127.16.21.62:45475)
0806 10:47:57.678810 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.678814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.679198 (+ 384us) server_negotiation.cc:407] Connection header received
0806 10:47:57.681016 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.681022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.681106 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.681207 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.683448 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.684579 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.687163 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.687579 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.687716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.697751 (+ 10035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.697760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.697768 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.697836 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.702289 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.703145 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.703161 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.703167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.703388 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.703816 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.703824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.703830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.704443 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.704682 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.705451 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.705808 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 10:47:57.706812 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.676984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.16.21.1:34847)
0806 10:47:57.677989 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.678037 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.680605 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.681471 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.681480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.682095 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.683156 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.683179 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.685056 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.685061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.687012 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.687026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.688207 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.688213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.688476 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.689449 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.689472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.694846 (+ 5374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.701011 (+ 6165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.701015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.701018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.701633 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.702089 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.702094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.702098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.702359 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.703077 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.703084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.703548 (+ 464us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.703881 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.705593 (+ 1712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":90,"thread_start_us":389,"threads_started":1}
I20260806 10:47:57.706939 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.679304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:58949 (local address 127.16.21.61:39685)
0806 10:47:57.679810 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.679817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.680847 (+ 1030us) server_negotiation.cc:407] Connection header received
0806 10:47:57.681035 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.681039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.681115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.681233 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.683678 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.686136 (+ 2458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.688112 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.688823 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.688958 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.696088 (+ 7130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.696095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.696099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.696139 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.700165 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.702436 (+ 2271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.702444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.702449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.702596 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.703085 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.703090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.703093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.703437 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.703584 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.706243 (+ 2659us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.706552 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260806 10:47:57.707113 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.678433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:59519)
0806 10:47:57.679084 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.679106 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.680605 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.681474 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.681481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.682465 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.683152 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.683172 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.684726 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.684731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.687019 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.687035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.687819 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.687826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.688130 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.689005 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.689037 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.696606 (+ 7569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.702425 (+ 5819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.702432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.702437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.702993 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.703505 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.703511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.703515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.703692 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.704761 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.704768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.704984 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.706367 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.706622 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":177,"thread_start_us":357,"threads_started":1}
I20260806 10:47:57.707268 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.679164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.16.21.1:58949)
0806 10:47:57.680742 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.680762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.680898 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.681474 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.681481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.682465 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:57.683506 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.683519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.686318 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.686329 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.687948 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.687962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.689275 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.689284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.689538 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.690175 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:57.690204 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:57.695884 (+ 5680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:57.700981 (+ 5097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.700990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.701006 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.702282 (+ 1276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.702732 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:57.702737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:57.702740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:57.702938 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:57.703665 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:57.703669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:57.703812 (+ 143us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.703953 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.704399 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"mutex_wait_us":133,"thread_start_us":1338,"threads_started":1}
I20260806 10:47:57.709174 23718 heartbeater.cc:347] Connected to a master server at 127.16.21.60:41973
I20260806 10:47:57.709226 23719 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45475
I20260806 10:47:57.709532 23718 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:57.709537 23719 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:57.710443 23718 heartbeater.cc:511] Master 127.16.21.60:41973 requested a full tablet report, sending...
I20260806 10:47:57.710705 23719 heartbeater.cc:511] Master 127.16.21.62:45475 requested a full tablet report, sending...
Aug 06 10:47:57 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.2@KRBTEST.COM
I20260806 10:47:57.711540 23720 heartbeater.cc:347] Connected to a master server at 127.16.21.61:39685
I20260806 10:47:57.711663 23720 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:57.712061 23720 heartbeater.cc:511] Master 127.16.21.61:39685 requested a full tablet report, sending...
I20260806 10:47:57.712971 23539 ts_manager.cc:194] Registered new tserver with Master: e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731)
I20260806 10:47:57.713991 23353 ts_manager.cc:194] Registered new tserver with Master: e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731)
I20260806 10:47:57.715101 22740 ts_manager.cc:194] Re-registered known tserver with Master: e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731)
I20260806 10:47:57.715729 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.691834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.16.21.60:41973)
0806 10:47:57.701468 (+ 9634us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.701474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.707825 (+ 6351us) server_negotiation.cc:407] Connection header received
0806 10:47:57.707897 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.707905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.708012 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.708140 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:57.708987 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.710189 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.712472 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.713814 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.713942 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.714183 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.715262 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.715393 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9492,"thread_start_us":9904,"threads_started":1}
I20260806 10:47:57.716009 22740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:59519
I20260806 10:47:57.715145 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.691625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.0.0.1:36384)
0806 10:47:57.705952 (+ 14327us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:57.705971 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:57.707380 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:57.708434 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:57.708440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:57.708471 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:57.708819 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.708833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.710368 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.710373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.712242 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:57.712286 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.714288 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.714294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:57.714528 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.714566 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:47:57.714650 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:47:57.714819 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14215,"thread_start_us":15497,"threads_started":1}
I20260806 10:47:57.717643 23549 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260806 10:47:57.718309 23577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece5d271795e4851b5e02b6f915b0710. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:57.718602 23577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: This master's current role is: LEARNER
I20260806 10:47:57.719825 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:57.684495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.16.21.2:45963)
0806 10:47:57.685200 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:57.685212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:57.685821 (+ 609us) server_negotiation.cc:407] Connection header received
0806 10:47:57.686030 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:57.686039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:57.686280 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:57.686399 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:57.691519 (+ 5120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.693579 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.696159 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:57.697129 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:57.697267 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:57.712144 (+ 14877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:57.712158 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:57.712173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:57.712216 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:57.716182 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.717160 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.717172 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.717176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.717349 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:57.717889 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:57.717893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:57.717896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:57.718311 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:57.718595 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:57.719195 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 10:47:57.719463 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":354,"threads_started":1}
I20260806 10:47:57.721055 23061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:57.804714 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22938
I20260806 10:47:57.824908 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:45963
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:45475
--webserver_port=34643
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:47:58.157861 23732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:58.158107 23732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:58.158391 23732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:58.165509 23732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:58.165613 23732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:58.165791 23732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:47:58.175438 23732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:45963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=34643
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:58.176646 23732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:58.177932 23732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:58.191260 23737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:58.191157 23738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:58.191300 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:58.192341 23732 server_base.cc:1034] running on GCE node
I20260806 10:47:58.192943 23732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:58.193877 23732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:58.195099 23732 hybrid_clock.cc:648] HybridClock initialized: now 1786013278195054 us; error 39 us; skew 500 ppm
I20260806 10:47:58.195416 23732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:58.198604 23732 init.cc:377] Logged in from keytab as oryx/127.16.21.2@KRBTEST.COM (short username oryx)
I20260806 10:47:58.200088 23732 webserver.cc:460] Webserver started at http://127.16.21.2:34643/ using document root <none> and password file <none>
I20260806 10:47:58.200856 23732 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:58.201017 23732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:58.206547 23732 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:47:58.209359 23747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:58.210628 23732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:47:58.210800 23732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "2f96449697744f509e9262d4e1ce283b"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "7b624a2cc655adfdd4932a15dc745648"
server_key_iv: "c6fd6f7809ecccfe735c65266436b5e0"
server_key_version: "encryptionkey@0"
I20260806 10:47:58.211781 23732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:58.231185 23732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:58.234827 23732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:58.235119 23732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:58.236073 23732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:58.237370 23732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:58.237493 23732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:58.237645 23732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:58.237735 23732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:58.267386 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: NON_VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: true } } }
I20260806 10:47:58.267690 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:58.270969 23849 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: attempting to promote NON_VOTER 7108e67ae9d0454b84f38a9095e42bb8 to VOTER
I20260806 10:47:58.271920 23849 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } }
I20260806 10:47:58.274515 23549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 LEARNER]: Refusing update from remote peer ece5d271795e4851b5e02b6f915b0710: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 10:47:58.274698 23363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Refusing update from remote peer ece5d271795e4851b5e02b6f915b0710: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 10:47:58.275452 23851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 10:47:58.275936 23849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 10:47:58.278728 23851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.281690 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.281945 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:58.282192 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.282440 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:58.282316 23363 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.288731 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.289096 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:47:58.289700 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.289964 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710 [sys.catalog]: This master's current role is: LEADER
I20260806 10:47:58.290361 23549 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.292066 23732 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:45963
I20260806 10:47:58.292126 23869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:45963 every 8 connection(s)
I20260806 10:47:58.293671 23732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:47:58.293861 23870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ece5d271795e4851b5e02b6f915b0710" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:47:58.294158 23870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:47:58.300720 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 23732
I20260806 10:47:58.301024 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 23071
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.60@KRBTEST.COM
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
I20260806 10:47:58.330401 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.299517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:57261 (local address 127.16.21.62:45475)
0806 10:47:58.300065 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.300071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.308075 (+ 8004us) server_negotiation.cc:407] Connection header received
0806 10:47:58.308576 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.308584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.308682 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.308793 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.310508 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.312097 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.315114 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.315820 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.315913 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.322545 (+ 6632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.322555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.322561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.322602 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.326474 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.327424 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.327432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.327438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.327599 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.328121 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.328128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.328134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.328552 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.328704 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.329690 (+ 986us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.329939 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":480,"threads_started":1}
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.3@KRBTEST.COM
I20260806 10:47:58.332680 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.299851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:57261)
0806 10:47:58.307957 (+ 8106us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.307979 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.308389 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.309069 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.309077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.309911 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:58.310299 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.310315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.312354 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.312362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.314921 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.314942 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.316415 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.316423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.316656 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.317589 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:58.317624 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:58.322286 (+ 4662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:58.326628 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.326637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.326650 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.327261 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.327746 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.327750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.327753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.327974 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.328807 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:58.328815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:58.329199 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 10:47:58.330680 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0806 10:47:58.331226 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7943,"mutex_wait_us":48,"spinlock_wait_cycles":1408,"thread_start_us":8242,"threads_started":1}
I20260806 10:47:58.334126 23872 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45475
I20260806 10:47:58.334474 23872 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:58.335357 23872 heartbeater.cc:511] Master 127.16.21.62:45475 requested a full tablet report, sending...
I20260806 10:47:58.335474 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.296842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:60227 (local address 127.16.21.61:39685)
0806 10:47:58.297335 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.297339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.298347 (+ 1008us) server_negotiation.cc:407] Connection header received
0806 10:47:58.301274 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.301280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.301375 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.301466 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.311878 (+ 10412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.313038 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.319205 (+ 6167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.319899 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.319993 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.325723 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.325730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.325735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.325772 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.329325 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.331239 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.331245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.331248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.331382 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.332162 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.332167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.332170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.332554 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.332669 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.334823 (+ 2154us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.335066 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260806 10:47:58.336452 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.297354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.16.21.2:60227)
0806 10:47:58.298209 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.298249 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.301131 (+ 2882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.308002 (+ 6871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.308014 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.308823 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:58.309616 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.309638 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.313271 (+ 3633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.313276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.317303 (+ 4027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.317319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.321207 (+ 3888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.321216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.321507 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.322183 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:58.322221 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:58.325529 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:58.329486 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.329490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.329493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.329980 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.331515 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.331520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.331523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.331700 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.332795 (+ 1095us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:58.332802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:58.332962 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 10:47:58.335661 (+ 2699us) client_negotiation.cc:240] Negotiation successful
0806 10:47:58.335969 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":2297,"spinlock_wait_cycles":3328,"thread_start_us":354,"threads_started":1}
I20260806 10:47:58.337481 23873 heartbeater.cc:347] Connected to a master server at 127.16.21.61:39685
I20260806 10:47:58.337613 23873 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:58.337682 22740 ts_manager.cc:194] Re-registered known tserver with Master: 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:47:58.338078 23873 heartbeater.cc:511] Master 127.16.21.61:39685 requested a full tablet report, sending...
I20260806 10:47:58.339361 22740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:57261
I20260806 10:47:58.340538 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.298601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.16.21.2:54745)
0806 10:47:58.299251 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.299283 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.300792 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.314710 (+ 13918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.314717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.315321 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:58.315595 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.315606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.317216 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.317222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.318885 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.318907 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.319954 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.319961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.320128 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.320749 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:58.320769 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:58.323922 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:58.336815 (+ 12893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.336820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.336823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.337347 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.338738 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.338743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.338747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.338924 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.339580 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:58.339585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:58.339720 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 10:47:58.339869 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 10:47:58.340159 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":290,"threads_started":1}
I20260806 10:47:58.340551 23353 ts_manager.cc:194] Registered new tserver with Master: 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:47:58.341265 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.298745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:54745 (local address 127.16.21.60:41973)
0806 10:47:58.314289 (+ 15544us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.314294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.314313 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:47:58.314365 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.314369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.314448 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.315744 (+ 1296us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.315928 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.317061 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.319099 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.319726 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.320466 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.324109 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.324115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.324119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.324159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.334557 (+ 10398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.338435 (+ 3878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.338440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.338444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.338599 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.339067 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.339073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.339078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.339429 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.340413 (+ 984us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.340549 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.340954 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15408,"thread_start_us":15792,"threads_started":1}
I20260806 10:47:58.342172 23871 heartbeater.cc:347] Connected to a master server at 127.16.21.60:41973
I20260806 10:47:58.342334 23871 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:58.342713 23871 heartbeater.cc:511] Master 127.16.21.60:41973 requested a full tablet report, sending...
I20260806 10:47:58.343233 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.302101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.16.21.3:41205)
0806 10:47:58.320016 (+ 17915us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.320025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.320052 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:47:58.320112 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.320119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.320338 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.322631 (+ 2293us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.322898 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.324465 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.326521 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.327372 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.327488 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.333871 (+ 6383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.333907 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.333921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.333974 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.338456 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.339415 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.339423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.339426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.340165 (+ 739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.340665 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.340669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.340672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.341115 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.341392 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.342583 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.342841 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17694,"thread_start_us":17722,"threads_started":1}
I20260806 10:47:58.343670 23539 ts_manager.cc:194] Registered new tserver with Master: 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:47:58.344435 23196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:58.427522 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 23071
I20260806 10:47:58.447003 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:41205
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:45475
--webserver_port=45795
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:47:58.718581 23719 heartbeater.cc:503] Master 127.16.21.62:45475 was elected leader, sending a full tablet report...
W20260806 10:47:58.778417 23884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:47:58.778684 23884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:47:58.778846 23884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:47:58.786139 23884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:47:58.786284 23884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:47:58.786455 23884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:47:58.796554 23884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:41205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=45795
--tserver_master_addrs=127.16.21.62:45475,127.16.21.61:39685,127.16.21.60:41973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:47:58.797896 23884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:47:58.799311 23884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:47:58.813364 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:58.813395 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:47:58.813489 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:47:58.814237 23884 server_base.cc:1034] running on GCE node
I20260806 10:47:58.814889 23884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:47:58.815889 23884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:47:58.817081 23884 hybrid_clock.cc:648] HybridClock initialized: now 1786013278817023 us; error 50 us; skew 500 ppm
I20260806 10:47:58.817400 23884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:47:58.820497 23884 init.cc:377] Logged in from keytab as oryx/127.16.21.3@KRBTEST.COM (short username oryx)
I20260806 10:47:58.821864 23884 webserver.cc:460] Webserver started at http://127.16.21.3:45795/ using document root <none> and password file <none>
I20260806 10:47:58.822517 23884 fs_manager.cc:359] Metadata directory not provided
I20260806 10:47:58.822682 23884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:47:58.827829 23884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:47:58.830399 23899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:58.831751 23884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:47:58.831939 23884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "fc046a7def484b08a6b2e2ae73728e04"
format_stamp: "Formatted at 2026-08-06 10:47:55 on dist-test-slave-0b58"
server_key: "b1038499cb58c8e448dc4b556df05fc0"
server_key_iv: "945d15fddbcc84a17115c23911b0300e"
server_key_version: "encryptionkey@0"
I20260806 10:47:58.832983 23884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:47:58.846753 23884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:47:58.850839 23884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:47:58.851115 23884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:47:58.851981 23884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:47:58.853331 23884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:47:58.853458 23884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:58.853585 23884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:47:58.853710 23884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:47:58.887636 23884 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41205
I20260806 10:47:58.887738 24013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41205 every 8 connection(s)
I20260806 10:47:58.889072 23884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:47:58.891804 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 23884
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.60@KRBTEST.COM
I20260806 10:47:58.919063 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.892067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:43029 (local address 127.16.21.60:41973)
0806 10:47:58.892616 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.892623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.893601 (+ 978us) server_negotiation.cc:407] Connection header received
0806 10:47:58.896969 (+ 3368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.896979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.897102 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.897268 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.899743 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.901269 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.903996 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.904716 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.904891 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.910327 (+ 5436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.910336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.910344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.910400 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.914789 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.915841 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.915849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.915855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.916055 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.916600 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.916607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.916613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.917060 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.917243 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.918208 (+ 965us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.918501 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":281,"threads_started":1}
I20260806 10:47:58.920356 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.892584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.16.21.3:43029)
0806 10:47:58.893413 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.893458 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.896620 (+ 3162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.897483 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.897495 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.898401 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:58.899523 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.899555 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.901423 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.901431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.903781 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.903799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.904973 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.904984 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.905410 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.906448 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:58.906490 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:58.910060 (+ 3570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:58.914931 (+ 4871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.914942 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.914962 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.915731 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.916179 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.916186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.916193 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.916491 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.917201 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:58.917208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:58.917660 (+ 452us) client_negotiation.cc:769] Sending connection context
0806 10:47:58.918069 (+ 409us) client_negotiation.cc:240] Negotiation successful
0806 10:47:58.918726 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":364,"threads_started":1}
I20260806 10:47:58.921968 24014 heartbeater.cc:347] Connected to a master server at 127.16.21.60:41973
I20260806 10:47:58.922309 24014 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:58.923380 24014 heartbeater.cc:511] Master 127.16.21.60:41973 requested a full tablet report, sending...
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.61@KRBTEST.COM
I20260806 10:47:58.925603 23539 ts_manager.cc:194] Registered new tserver with Master: fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205)
Aug 06 10:47:58 dist-test-slave-0b58 krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013278, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:47:58.936743 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.897513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.16.21.3:58797)
0806 10:47:58.901118 (+ 3605us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.901142 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.901252 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.908319 (+ 7067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.908324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.908872 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:58.910540 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.910555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.912636 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.912645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.914667 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.914681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.921575 (+ 6894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.921581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.921733 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.922641 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:58.922676 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:58.926342 (+ 3666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:58.932870 (+ 6528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.932876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.932879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.934238 (+ 1359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.934744 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.934751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.934755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.934932 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.935647 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:58.935652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:58.935810 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 10:47:58.935952 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 10:47:58.936249 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3486,"thread_start_us":3338,"threads_started":1}
I20260806 10:47:58.938354 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.897732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:58797 (local address 127.16.21.61:39685)
0806 10:47:58.904410 (+ 6678us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.904418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.904443 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:47:58.904522 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.904530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.904642 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.910708 (+ 6066us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.910917 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.912433 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.915907 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.921175 (+ 5268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.921330 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.928597 (+ 7267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.928606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.928614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.928664 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.932665 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.934415 (+ 1750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.934422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.934428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.934598 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.935110 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.935116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.935121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.935492 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.937078 (+ 1586us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.937219 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.937440 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6492,"thread_start_us":6359,"threads_started":1}
I20260806 10:47:58.939113 24016 heartbeater.cc:347] Connected to a master server at 127.16.21.61:39685
I20260806 10:47:58.939261 24016 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:58.939769 24016 heartbeater.cc:511] Master 127.16.21.61:39685 requested a full tablet report, sending...
I20260806 10:47:58.940945 23353 ts_manager.cc:194] Registered new tserver with Master: fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205)
I20260806 10:47:58.945004 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.894016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:45197)
0806 10:47:58.900047 (+ 6031us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.900070 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.900192 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.924532 (+ 24340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.924539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.925104 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:47:58.925528 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.925545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.929647 (+ 4102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.929655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.931626 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.931643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.932925 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.932936 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:58.933134 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.933725 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:47:58.933741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:47:58.936760 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:47:58.941053 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.941058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.941061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.941565 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.942044 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:47:58.942049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:47:58.942052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:47:58.942237 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:47:58.943946 (+ 1709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:47:58.943953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:47:58.944123 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 10:47:58.944293 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 10:47:58.944571 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5866,"thread_start_us":3163,"threads_started":1}
I20260806 10:47:58.946120 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.894205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:45197 (local address 127.16.21.62:45475)
0806 10:47:58.922118 (+ 27913us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.922128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.922155 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:47:58.922234 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.922242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.922357 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.926555 (+ 4198us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.926786 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.928357 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.931905 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.932699 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.933438 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.936958 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.936965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.936969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.937005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.940904 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.941732 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.941737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.941740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.941915 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.942885 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.942890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.942894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.943217 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.943351 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.945413 (+ 2062us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.945644 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22860,"thread_start_us":22755,"threads_started":1}
I20260806 10:47:58.946962 24015 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45475
I20260806 10:47:58.947093 24015 heartbeater.cc:464] Registering TS with master...
I20260806 10:47:58.947520 24015 heartbeater.cc:511] Master 127.16.21.62:45475 requested a full tablet report, sending...
I20260806 10:47:58.948544 22740 ts_manager.cc:194] Re-registered known tserver with Master: fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205)
I20260806 10:47:58.949609 22740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:45197
I20260806 10:47:58.951824 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.912045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.16.21.61:39685)
0806 10:47:58.921349 (+ 9304us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.921355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.925777 (+ 4422us) server_negotiation.cc:407] Connection header received
0806 10:47:58.926749 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.926757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.926854 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.926972 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.930162 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.931510 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.935944 (+ 4434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.936651 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.937029 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.943974 (+ 6945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.943981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.943985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.944022 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.947844 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.948809 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.948817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.948820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.948977 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.949454 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.949459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.949464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.949794 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.950985 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.951142 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.951366 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8397,"thread_start_us":8263,"threads_started":1}
I20260806 10:47:58.953816 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.905357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.16.21.62:45475)
0806 10:47:58.929517 (+ 24160us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.933190 (+ 3673us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.933211 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:47:58.933264 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.933268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.933362 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.933472 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.935997 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.937363 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.940353 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.941060 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.941187 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.944808 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.944814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.944819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.944854 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.949116 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.949848 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.949853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.949856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.950236 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.951391 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.951396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.951400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.951790 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.952002 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.953408 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.953573 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24012,"thread_start_us":27715,"threads_started":1}
I20260806 10:47:58.957526 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.925805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.16.21.60:41973)
0806 10:47:58.925951 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.925956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.932807 (+ 6851us) server_negotiation.cc:407] Connection header received
0806 10:47:58.933688 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.933695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.933812 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.933917 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:58.940141 (+ 6224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.941127 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.943260 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:58.943979 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:58.944110 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:58.950062 (+ 5952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:58.950068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:58.950072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:58.950118 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:58.954441 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.955253 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.955260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.955268 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.955392 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:58.955855 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:58.955862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:58.955867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:58.956249 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:58.956538 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:58.956802 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 10:47:58.957026 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 10:47:58.958076 22740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:47:58.965051 22740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:47:59.006233 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.980547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34731 (local address 127.0.0.1:60446)
0806 10:47:58.981155 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.981175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.981315 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:58.999128 (+ 17813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:58.999132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:58.999154 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:58.999604 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:58.999618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.001654 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.001663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.003308 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.003342 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.005332 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.005342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.005506 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.005540 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:47:59.005634 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 10:47:59.005766 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":335,"threads_started":1}
I20260806 10:47:59.007058 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.982080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45963 (local address 127.0.0.1:58642)
0806 10:47:58.998072 (+ 15992us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.998092 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.999410 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:59.000062 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:59.000067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:59.000108 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:59.000580 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.000597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.002406 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.002416 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.004425 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.004438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.006242 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.006251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.006474 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.006511 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 10:47:59.006595 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:47:59.006732 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15830,"thread_start_us":15674,"threads_started":1}
I20260806 10:47:59.007750 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.982663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.16.21.2:45963)
0806 10:47:58.996808 (+ 14145us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.996821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.998213 (+ 1392us) server_negotiation.cc:407] Connection header received
0806 10:47:58.999584 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.999592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.999870 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.000071 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:59.000775 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.002236 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.004614 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.005728 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.005844 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.006041 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.007286 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.007464 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13936,"thread_start_us":13797,"threads_started":1}
I20260806 10:47:59.008515 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.981106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.16.21.1:34731)
0806 10:47:58.998658 (+ 17552us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.998671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.998691 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:47:58.998769 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.998774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.998964 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:58.999112 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:58.999835 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.001519 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.003566 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.005080 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.005215 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.005428 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.008052 (+ 2624us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.008216 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17392,"thread_start_us":17283,"threads_started":1}
I20260806 10:47:59.009994 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.981766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.16.21.3:41205)
0806 10:47:58.994537 (+ 12771us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:58.994549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:58.997559 (+ 3010us) server_negotiation.cc:407] Connection header received
0806 10:47:58.999706 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:58.999711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:58.999957 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.000192 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:59.001739 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.003256 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.006565 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.008957 (+ 2392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.009090 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.009265 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.009595 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.009734 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12552,"thread_start_us":12774,"threads_started":1}
I20260806 10:47:59.010044 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:58.980990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41205 (local address 127.0.0.1:50764)
0806 10:47:58.997392 (+ 16402us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:58.997426 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:58.999530 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:59.000374 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:59.000382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:59.000414 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:59.000835 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.000844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.003440 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.003445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.005938 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.005951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.009181 (+ 3230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.009188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.009378 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.009410 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:47:59.009495 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 10:47:59.009626 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16230,"mutex_wait_us":24,"thread_start_us":16029,"threads_started":1}
I20260806 10:47:59.011643 23796 tablet_service.cc:1467] Processing CreateTablet for tablet f6b3d39a93124455bea5d6acfcb759cb (DEFAULT_TABLE table=test-table [id=0936605ed8a349ebb8826c3ad0b5b7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:59.013042 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b3d39a93124455bea5d6acfcb759cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:59.014251 23653 tablet_service.cc:1467] Processing CreateTablet for tablet f6b3d39a93124455bea5d6acfcb759cb (DEFAULT_TABLE table=test-table [id=0936605ed8a349ebb8826c3ad0b5b7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:59.015408 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b3d39a93124455bea5d6acfcb759cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:59.015414 23948 tablet_service.cc:1467] Processing CreateTablet for tablet f6b3d39a93124455bea5d6acfcb759cb (DEFAULT_TABLE table=test-table [id=0936605ed8a349ebb8826c3ad0b5b7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:47:59.016604 23948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b3d39a93124455bea5d6acfcb759cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:47:59.038044 24047 tablet_bootstrap.cc:492] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Bootstrap starting.
I20260806 10:47:59.040974 24048 tablet_bootstrap.cc:492] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Bootstrap starting.
I20260806 10:47:59.041165 24047 tablet_bootstrap.cc:654] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:59.042524 24047 log.cc:824] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:59.043952 24049 tablet_bootstrap.cc:492] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Bootstrap starting.
I20260806 10:47:59.045055 24048 tablet_bootstrap.cc:654] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:59.046689 24048 log.cc:824] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:59.046886 24049 tablet_bootstrap.cc:654] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Neither blocks nor log segments found. Creating new log.
I20260806 10:47:59.048084 24049 log.cc:824] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Log is configured to *not* fsync() on all Append() calls
I20260806 10:47:59.050063 24048 tablet_bootstrap.cc:492] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: No bootstrap required, opened a new log
I20260806 10:47:59.050323 24048 ts_tablet_manager.cc:1397] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 10:47:59.050954 24049 tablet_bootstrap.cc:492] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: No bootstrap required, opened a new log
I20260806 10:47:59.051203 24049 ts_tablet_manager.cc:1397] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 10:47:59.051362 24047 tablet_bootstrap.cc:492] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: No bootstrap required, opened a new log
I20260806 10:47:59.051601 24047 ts_tablet_manager.cc:1397] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260806 10:47:59.058961 24049 raft_consensus.cc:348] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.059373 24049 raft_consensus.cc:374] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:59.059499 24049 raft_consensus.cc:729] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f96449697744f509e9262d4e1ce283b, State: Initialized, Role: FOLLOWER
I20260806 10:47:59.060096 24049 consensus_queue.cc:260] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.061092 24048 raft_consensus.cc:348] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.061496 24048 raft_consensus.cc:374] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:59.061590 24048 raft_consensus.cc:729] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f868ef8cfc45de8c64dcbb6c79d7ce, State: Initialized, Role: FOLLOWER
I20260806 10:47:59.061766 24049 ts_tablet_manager.cc:1428] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Time spent starting tablet: real 0.010s user 0.010s sys 0.002s
I20260806 10:47:59.062036 23872 heartbeater.cc:503] Master 127.16.21.62:45475 was elected leader, sending a full tablet report...
I20260806 10:47:59.062217 24048 consensus_queue.cc:260] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.062395 24047 raft_consensus.cc:348] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.062820 24047 raft_consensus.cc:374] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:47:59.062935 24047 raft_consensus.cc:729] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc046a7def484b08a6b2e2ae73728e04, State: Initialized, Role: FOLLOWER
I20260806 10:47:59.063627 24047 consensus_queue.cc:260] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.069284 24048 ts_tablet_manager.cc:1428] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260806 10:47:59.069510 24047 ts_tablet_manager.cc:1428] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260806 10:47:59.069902 24015 heartbeater.cc:503] Master 127.16.21.62:45475 was elected leader, sending a full tablet report...
W20260806 10:47:59.142113 24017 tablet.cc:2369] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:47:59.178158 23721 tablet.cc:2369] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:59.276515 24055 raft_consensus.cc:482] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:47:59.276813 24055 raft_consensus.cc:504] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.278425 24055 leader_election.cc:288] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731), 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:47:59.284125 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.278758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:40631 (local address 127.16.21.1:34731)
0806 10:47:59.278980 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.278987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.279010 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:47:59.279259 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.279265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.279345 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.279437 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:59.279826 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.280661 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.282356 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.283142 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.283284 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.283389 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.283719 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.283853 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:47:59.284238 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.279305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:44545 (local address 127.16.21.2:45963)
0806 10:47:59.279518 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.279523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.279537 (+ 14us) server_negotiation.cc:407] Connection header received
0806 10:47:59.279594 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.279597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.279649 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.279749 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:47:59.280103 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.280871 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.282512 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.283288 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.283427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.283548 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.283840 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.283962 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:47:59.284168 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.278652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34731 (local address 127.16.21.3:40631)
0806 10:47:59.278914 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:59.278937 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:59.279070 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:59.279456 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:59.279460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:59.279483 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:59.279665 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.279673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.280815 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.280821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.282168 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.282180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.283319 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.283325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.283538 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.283561 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:47:59.283688 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 10:47:59.283812 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":1}
I20260806 10:47:59.284363 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.279055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45963 (local address 127.16.21.3:44545)
0806 10:47:59.279219 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 10:47:59.279233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:47:59.279322 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:47:59.279765 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:47:59.279769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:47:59.279793 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:47:59.279966 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.279974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.281008 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.281013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.282354 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:47:59.282367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.283470 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.283476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:47:59.283690 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.283712 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 10:47:59.283815 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 10:47:59.283949 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 10:47:59.285557 23817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b3d39a93124455bea5d6acfcb759cb" candidate_uuid: "fc046a7def484b08a6b2e2ae73728e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f96449697744f509e9262d4e1ce283b" is_pre_election: true
I20260806 10:47:59.285626 23673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b3d39a93124455bea5d6acfcb759cb" candidate_uuid: "fc046a7def484b08a6b2e2ae73728e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" is_pre_election: true
I20260806 10:47:59.286119 23817 raft_consensus.cc:2454] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc046a7def484b08a6b2e2ae73728e04 in term 0.
I20260806 10:47:59.286175 23673 raft_consensus.cc:2454] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc046a7def484b08a6b2e2ae73728e04 in term 0.
I20260806 10:47:59.286892 23903 leader_election.cc:302] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7f868ef8cfc45de8c64dcbb6c79d7ce, fc046a7def484b08a6b2e2ae73728e04; no voters:
I20260806 10:47:59.287214 24055 raft_consensus.cc:2781] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:47:59.287381 24055 raft_consensus.cc:482] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:47:59.287505 24055 raft_consensus.cc:3037] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:59.289853 24055 raft_consensus.cc:504] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
I20260806 10:47:59.290405 24055 leader_election.cc:288] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [CANDIDATE]: Term 1 election: Requested vote from peers e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731), 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:47:59.290822 23673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b3d39a93124455bea5d6acfcb759cb" candidate_uuid: "fc046a7def484b08a6b2e2ae73728e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce"
I20260806 10:47:59.290894 23817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b3d39a93124455bea5d6acfcb759cb" candidate_uuid: "fc046a7def484b08a6b2e2ae73728e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f96449697744f509e9262d4e1ce283b"
I20260806 10:47:59.291004 23673 raft_consensus.cc:3037] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:59.291107 23817 raft_consensus.cc:3037] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:47:59.293704 23673 raft_consensus.cc:2454] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc046a7def484b08a6b2e2ae73728e04 in term 1.
I20260806 10:47:59.294054 23903 leader_election.cc:302] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7f868ef8cfc45de8c64dcbb6c79d7ce, fc046a7def484b08a6b2e2ae73728e04; no voters:
I20260806 10:47:59.294365 24055 raft_consensus.cc:2781] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:47:59.294561 23817 raft_consensus.cc:2454] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc046a7def484b08a6b2e2ae73728e04 in term 1.
I20260806 10:47:59.294965 24055 raft_consensus.cc:686] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 1 LEADER]: Becoming Leader. State: Replica: fc046a7def484b08a6b2e2ae73728e04, State: Running, Role: LEADER
I20260806 10:47:59.295727 24055 consensus_queue.cc:237] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } }
W20260806 10:47:59.297729 23874 tablet.cc:2369] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:47:59.301645 22741 catalog_manager.cc:5495] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "fc046a7def484b08a6b2e2ae73728e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } health_report { overall_health: UNKNOWN } } }
I20260806 10:47:59.368616 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.363323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.16.21.3:41205)
0806 10:47:59.363507 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.363513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.363610 (+ 97us) server_negotiation.cc:407] Connection header received
0806 10:47:59.363844 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.363848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.363922 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.364069 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:47:59.364484 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.365152 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.366480 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.366901 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.367032 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.367523 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:47:59.368018 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:47:59.368196 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.368252 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.368366 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:47:59.376123 23817 raft_consensus.cc:1261] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Refusing update from remote peer fc046a7def484b08a6b2e2ae73728e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:59.376152 23673 raft_consensus.cc:1261] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Refusing update from remote peer fc046a7def484b08a6b2e2ae73728e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:47:59.377092 24056 consensus_queue.cc:1035] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:59.377467 24055 consensus_queue.cc:1035] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:47:59.391597 24059 mvcc.cc:205] Tried to move back new op lower bound from 7315510392315506688 to 7315510391999483904. Current Snapshot: MvccSnapshot[applied={T|T < 7315510392315506688}]
I20260806 10:47:59.414119 22741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:59.414376 22741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:47:59.420845 22741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ece5d271795e4851b5e02b6f915b0710: Sending DeleteTablet for 3 replicas of tablet f6b3d39a93124455bea5d6acfcb759cb
I20260806 10:47:59.422123 23653 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b3d39a93124455bea5d6acfcb759cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:59 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:60446
I20260806 10:47:59.422436 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b3d39a93124455bea5d6acfcb759cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:59 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:58642
I20260806 10:47:59.423422 24069 tablet_replica.cc:330] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: stopping tablet replica
I20260806 10:47:59.423867 24069 raft_consensus.cc:2229] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:59.424067 24070 tablet_replica.cc:330] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: stopping tablet replica
I20260806 10:47:59.424371 24069 raft_consensus.cc:2258] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:59.424480 24070 raft_consensus.cc:2229] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:47:59.424764 23948 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b3d39a93124455bea5d6acfcb759cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:47:59 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:50764
I20260806 10:47:59.424947 24070 raft_consensus.cc:2258] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:59.427493 24072 tablet_replica.cc:330] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: stopping tablet replica
I20260806 10:47:59.427528 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 22709
I20260806 10:47:59.427548 24069 ts_tablet_manager.cc:1905] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:59.428032 24070 ts_tablet_manager.cc:1905] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:59.428483 24072 raft_consensus.cc:2229] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:47:59.428552 22771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:47:59.429378 24072 raft_consensus.cc:2258] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:47:59.432943 24072 ts_tablet_manager.cc:1905] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:47:59.435045 24070 ts_tablet_manager.cc:1918] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:59.435210 24070 log.cc:1196] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/f6b3d39a93124455bea5d6acfcb759cb
I20260806 10:47:59.435755 24070 ts_tablet_manager.cc:1939] T f6b3d39a93124455bea5d6acfcb759cb P 2f96449697744f509e9262d4e1ce283b: Deleting consensus metadata
I20260806 10:47:59.440397 24069 ts_tablet_manager.cc:1918] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:59.440562 24069 log.cc:1196] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/f6b3d39a93124455bea5d6acfcb759cb
I20260806 10:47:59.441236 24069 ts_tablet_manager.cc:1939] T f6b3d39a93124455bea5d6acfcb759cb P e7f868ef8cfc45de8c64dcbb6c79d7ce: Deleting consensus metadata
I20260806 10:47:59.446971 24072 ts_tablet_manager.cc:1918] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:47:59.448904 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.424451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.16.21.61:39685)
0806 10:47:59.424656 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.424663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.424684 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:47:59.424740 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.424745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.424824 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.424923 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:59.430300 (+ 5377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.431503 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.434109 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.434867 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.434984 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.441869 (+ 6885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:59.441876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:59.441881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:59.441921 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:59.445719 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.446564 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.446570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.446574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.446724 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.447145 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.447150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.447154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.447595 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:59.447851 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.448218 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.448557 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:47:59.448021 24072 log.cc:1196] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/f6b3d39a93124455bea5d6acfcb759cb
I20260806 10:47:59.450984 24072 ts_tablet_manager.cc:1939] T f6b3d39a93124455bea5d6acfcb759cb P fc046a7def484b08a6b2e2ae73728e04: Deleting consensus metadata
I20260806 10:47:59.465299 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.425929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.16.21.60:41973)
0806 10:47:59.426083 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.426090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.448973 (+ 22883us) server_negotiation.cc:407] Connection header received
0806 10:47:59.449223 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.449229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.449321 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.449395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:59.451505 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.452490 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.454225 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.454852 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.454985 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.458509 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:59.458516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:59.458522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:59.458563 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:59.462226 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.463131 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.463137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.463141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.463300 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.463703 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.463709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.463712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.464089 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:59.464372 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.464696 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.464919 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 10:47:59.569036 22726 catalog_manager.cc:4841] TS fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205): tablet f6b3d39a93124455bea5d6acfcb759cb (table test-table [id=0936605ed8a349ebb8826c3ad0b5b7a8]) successfully deleted
I20260806 10:47:59.569456 22728 catalog_manager.cc:4841] TS 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963): tablet f6b3d39a93124455bea5d6acfcb759cb (table test-table [id=0936605ed8a349ebb8826c3ad0b5b7a8]) successfully deleted
W20260806 10:47:59.570499 22728 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.21.2:45963, user_credentials={real_user=slave}} blocked reactor thread for 132072us
I20260806 10:47:59.570894 22728 catalog_manager.cc:4841] TS e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731): tablet f6b3d39a93124455bea5d6acfcb759cb (table test-table [id=0936605ed8a349ebb8826c3ad0b5b7a8]) successfully deleted
I20260806 10:47:59.571455 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 22709
I20260806 10:47:59.628494 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.612216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.16.21.60:41973)
0806 10:47:59.612457 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.612463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.612992 (+ 529us) server_negotiation.cc:407] Connection header received
0806 10:47:59.613130 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.613136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.613216 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.613325 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:59.615440 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.616410 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.618458 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.619092 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.619244 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.621802 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:59.621808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:59.621813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:59.621857 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:59.625382 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.626108 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.626114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.626117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.626288 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.626716 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.626721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.626724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.627058 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:59.627260 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.627874 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.628095 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 10:47:59.629808 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:47:59.611449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.16.21.61:39685)
0806 10:47:59.611629 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:47:59.611636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:47:59.612080 (+ 444us) server_negotiation.cc:407] Connection header received
0806 10:47:59.612377 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:47:59.612384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:47:59.612485 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:47:59.612573 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:47:59.614914 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.616178 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.617892 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:47:59.618556 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:47:59.618672 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:47:59.622327 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:47:59.622336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:47:59.622344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:47:59.622386 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:47:59.626500 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.627146 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.627153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.627156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.627282 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:47:59.627710 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:47:59.627714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:47:59.627718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:47:59.628073 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:47:59.628333 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 10:47:59.629255 (+ 922us) server_negotiation.cc:299] Negotiation successful
0806 10:47:59.629493 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:48:00.405047 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.404166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:59513)
0806 10:48:00.404663 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.404839 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
W20260806 10:48:00.405654 24015 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:45475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.406680 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.405808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:51751)
0806 10:48:00.406345 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.406473 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":254,"threads_started":1}
W20260806 10:48:00.407048 23872 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:45475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.412923 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.411011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:56635)
0806 10:48:00.412625 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.412769 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":1397,"threads_started":1}
W20260806 10:48:00.413326 23719 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:45475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.921053 24083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ece5d271795e4851b5e02b6f915b0710)
I20260806 10:48:00.921342 24083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } }
I20260806 10:48:00.922144 24084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ece5d271795e4851b5e02b6f915b0710)
I20260806 10:48:00.922389 24084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } }
I20260806 10:48:00.922878 24083 leader_election.cc:288] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475), 4525dcf585fa4701b6c7623f391e956f (127.16.21.61:39685)
I20260806 10:48:00.924034 24084 leader_election.cc:288] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475), 7108e67ae9d0454b84f38a9095e42bb8 (127.16.21.60:41973)
I20260806 10:48:00.924989 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.922953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60004)
0806 10:48:00.923380 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.923610 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
W20260806 10:48:00.925263 23526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.926534 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.924008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60020)
0806 10:48:00.926123 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.926340 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1014,"thread_start_us":878,"threads_started":1}
W20260806 10:48:00.926862 23340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.929576 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.929116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60034)
0806 10:48:00.929329 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.929421 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 10:48:00.929651 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.929106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60048)
0806 10:48:00.929394 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.929484 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260806 10:48:00.929818 23340 leader_election.cc:334] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475): Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
W20260806 10:48:00.929751 23526 leader_election.cc:334] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475): Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.932886 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.923627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39685 (local address 127.0.0.1:42294)
0806 10:48:00.924957 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.924978 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:00.925131 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:00.925675 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:00.925682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:00.925721 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:00.927068 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:00.927089 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.928529 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.928537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:00.930897 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:00.930912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.932346 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.932350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:00.932466 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:00.932489 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:48:00.932565 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:48:00.932644 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":1,"thread_start_us":991,"threads_started":1}
I20260806 10:48:00.933100 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.923714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.16.21.61:39685)
0806 10:48:00.924684 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:00.924690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:00.925161 (+ 471us) server_negotiation.cc:407] Connection header received
0806 10:48:00.925244 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:00.925251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:00.925337 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:00.925446 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:00.927327 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.928313 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.931040 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.932157 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.932305 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:00.932398 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:00.932609 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 10:48:00.932735 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":837,"thread_start_us":612,"threads_started":1}
I20260806 10:48:00.933724 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7108e67ae9d0454b84f38a9095e42bb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4525dcf585fa4701b6c7623f391e956f" is_pre_election: true
I20260806 10:48:00.934029 23363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7108e67ae9d0454b84f38a9095e42bb8 in term 1.
I20260806 10:48:00.934518 23526 leader_election.cc:302] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4525dcf585fa4701b6c7623f391e956f, 7108e67ae9d0454b84f38a9095e42bb8; no voters: ece5d271795e4851b5e02b6f915b0710
I20260806 10:48:00.934875 24083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 10:48:00.935016 24083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ece5d271795e4851b5e02b6f915b0710)
I20260806 10:48:00.935170 24083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:48:00.935832 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.927020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.16.21.60:41973)
0806 10:48:00.929204 (+ 2184us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:00.929210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:00.929234 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:48:00.929308 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:00.929313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:00.929386 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:00.930121 (+ 735us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:00.930428 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.931411 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.933659 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.934716 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.934878 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:00.934985 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:00.935338 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 10:48:00.935464 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2023,"thread_start_us":1818,"threads_started":1}
I20260806 10:48:00.935905 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.925250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41973 (local address 127.0.0.1:43612)
0806 10:48:00.926005 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.926024 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:00.926133 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:00.929540 (+ 3407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:00.929546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:00.929578 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:00.930209 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:00.930220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.931529 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.931534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:00.933504 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:00.933518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:00.934907 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:00.934913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:00.935173 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:00.935197 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:48:00.935310 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 10:48:00.935430 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":19,"thread_start_us":415,"threads_started":1}
I20260806 10:48:00.936503 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4525dcf585fa4701b6c7623f391e956f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7108e67ae9d0454b84f38a9095e42bb8" is_pre_election: true
I20260806 10:48:00.937848 24083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } }
I20260806 10:48:00.938305 23549 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4525dcf585fa4701b6c7623f391e956f in current term 2: Already voted for candidate 7108e67ae9d0454b84f38a9095e42bb8 in this term.
I20260806 10:48:00.938499 24083 leader_election.cc:288] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [CANDIDATE]: Term 2 election: Requested vote from peers ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475), 4525dcf585fa4701b6c7623f391e956f (127.16.21.61:39685)
I20260806 10:48:00.938980 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.938544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60050)
0806 10:48:00.938758 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.938852 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 10:48:00.939088 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7108e67ae9d0454b84f38a9095e42bb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4525dcf585fa4701b6c7623f391e956f"
I20260806 10:48:00.939288 23363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:48:00.941349 23363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7108e67ae9d0454b84f38a9095e42bb8 in term 2.
I20260806 10:48:00.941839 23526 leader_election.cc:302] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4525dcf585fa4701b6c7623f391e956f, 7108e67ae9d0454b84f38a9095e42bb8; no voters:
I20260806 10:48:00.941892 23340 leader_election.cc:302] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4525dcf585fa4701b6c7623f391e956f; no voters: 7108e67ae9d0454b84f38a9095e42bb8, ece5d271795e4851b5e02b6f915b0710
I20260806 10:48:00.942250 24084 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:48:00.942441 24083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:48:00.942657 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.942229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60064)
0806 10:48:00.942403 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.942513 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 10:48:00.942842 23526 leader_election.cc:334] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475): Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
I20260806 10:48:00.943161 24083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [term 2 LEADER]: Becoming Leader. State: Replica: 7108e67ae9d0454b84f38a9095e42bb8, State: Running, Role: LEADER
I20260806 10:48:00.943475 24083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } }
I20260806 10:48:00.945204 24093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7108e67ae9d0454b84f38a9095e42bb8. Latest consensus state: current_term: 2 leader_uuid: "7108e67ae9d0454b84f38a9095e42bb8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:48:00.945554 24093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:00.946206 24095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:00.949965 24095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:00.950783 24095 catalog_manager.cc:1259] Loaded cluster ID: 2a75c74a8c5b42cdaa63240d0a22fb29
I20260806 10:48:00.950850 24095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:00.951988 24095 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:00.952795 24095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Loaded TSK: 0
I20260806 10:48:00.953558 24095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:48:00.978626 23538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:48:00.991573 23538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:00.996376 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.995802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60092)
0806 10:48:00.996025 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.996130 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 10:48:00.996546 23363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [term 2 FOLLOWER]: Refusing update from remote peer 7108e67ae9d0454b84f38a9095e42bb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 10:48:00.997478 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:00.997110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60098)
0806 10:48:00.997263 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:00.997326 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:48:00.997704 24093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 10:48:00.998250 23526 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 -> Peer ece5d271795e4851b5e02b6f915b0710 (127.16.21.62:45475): Couldn't send request to peer ece5d271795e4851b5e02b6f915b0710. Status: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 10:48:01.000540 24084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7108e67ae9d0454b84f38a9095e42bb8. Latest consensus state: current_term: 2 leader_uuid: "7108e67ae9d0454b84f38a9095e42bb8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:48:01.000890 24084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:48:01.002712 24093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7108e67ae9d0454b84f38a9095e42bb8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:48:01.003003 24093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:01.009280 24084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7108e67ae9d0454b84f38a9095e42bb8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ece5d271795e4851b5e02b6f915b0710" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45475 } } peers { permanent_uuid: "4525dcf585fa4701b6c7623f391e956f" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39685 } attrs { promote: false } } peers { permanent_uuid: "7108e67ae9d0454b84f38a9095e42bb8" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41973 } attrs { promote: false } } }
I20260806 10:48:01.009564 24084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525dcf585fa4701b6c7623f391e956f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:48:01.022322 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.015309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41205 (local address 127.0.0.1:60770)
0806 10:48:01.015472 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.015490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:01.015594 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:01.016566 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:01.016571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:01.016596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:01.016790 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.016802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.018484 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.018490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.020036 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.020047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.021544 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.021549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.021772 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.021796 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:48:01.021910 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 10:48:01.022042 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 10:48:01.022413 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.015637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.16.21.3:41205)
0806 10:48:01.016218 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.016222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.016236 (+ 14us) server_negotiation.cc:407] Connection header received
0806 10:48:01.016375 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.016379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.016433 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.016534 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:01.016969 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.018340 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.020194 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.021389 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.021537 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.021657 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.021944 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.022075 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":314,"threads_started":1}
I20260806 10:48:01.023272 23948 tablet_service.cc:1467] Processing CreateTablet for tablet c0170f2d5f0b4ca4b8649e2a76c2c614 (DEFAULT_TABLE table=test-table [id=fdda3e548ed34ee6b6be2756b74c785a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:01.023805 23948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0170f2d5f0b4ca4b8649e2a76c2c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:01.033006 24105 tablet_bootstrap.cc:492] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Bootstrap starting.
I20260806 10:48:01.034965 24105 tablet_bootstrap.cc:654] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:01.041074 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.015747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45963 (local address 127.0.0.1:51888)
0806 10:48:01.016091 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.016111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:01.016236 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:01.033097 (+ 16861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:01.033104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:01.033137 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:01.033518 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.033532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.035188 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.035196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.037808 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.037827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.039964 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.039973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.040320 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.040373 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 10:48:01.040464 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:48:01.040663 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":32}
I20260806 10:48:01.041534 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.018768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34731 (local address 127.0.0.1:35460)
0806 10:48:01.019334 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.019354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:01.019469 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:01.029815 (+ 10346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:01.029819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:01.029845 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:01.030274 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.030290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.033233 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.033238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.037655 (+ 4417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.037683 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.039917 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.039928 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.040248 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.040900 (+ 652us) client_negotiation.cc:769] Sending connection context
0806 10:48:01.041004 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 10:48:01.041147 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":1174,"thread_start_us":575,"threads_started":1}
I20260806 10:48:01.041790 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.019175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.16.21.1:34731)
0806 10:48:01.028892 (+ 9717us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.028901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.028927 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:48:01.029019 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.029028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.029127 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.029248 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:01.030709 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.032206 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.037932 (+ 5726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.039526 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.040381 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.040553 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.041045 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.041459 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9592,"thread_start_us":9632,"threads_started":1}
I20260806 10:48:01.042640 23653 tablet_service.cc:1467] Processing CreateTablet for tablet c0170f2d5f0b4ca4b8649e2a76c2c614 (DEFAULT_TABLE table=test-table [id=fdda3e548ed34ee6b6be2756b74c785a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:01.042781 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.016554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.16.21.2:45963)
0806 10:48:01.032694 (+ 16140us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.032702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.032722 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:48:01.032807 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.032812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.032892 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.032998 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:01.033715 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.035004 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.038100 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.039675 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.041728 (+ 2053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.041841 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.042306 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.042461 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15988,"thread_start_us":15862,"threads_started":1}
I20260806 10:48:01.043146 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0170f2d5f0b4ca4b8649e2a76c2c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:01.043495 23796 tablet_service.cc:1467] Processing CreateTablet for tablet c0170f2d5f0b4ca4b8649e2a76c2c614 (DEFAULT_TABLE table=test-table [id=fdda3e548ed34ee6b6be2756b74c785a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:01.043864 24105 tablet_bootstrap.cc:492] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: No bootstrap required, opened a new log
I20260806 10:48:01.043941 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0170f2d5f0b4ca4b8649e2a76c2c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:01.044034 24105 ts_tablet_manager.cc:1397] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260806 10:48:01.045089 24105 raft_consensus.cc:348] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.045380 24105 raft_consensus.cc:374] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:01.045472 24105 raft_consensus.cc:729] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc046a7def484b08a6b2e2ae73728e04, State: Initialized, Role: FOLLOWER
I20260806 10:48:01.045730 24105 consensus_queue.cc:260] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.055802 24108 tablet_bootstrap.cc:492] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Bootstrap starting.
I20260806 10:48:01.056668 24105 ts_tablet_manager.cc:1428] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Time spent starting tablet: real 0.012s user 0.002s sys 0.011s
I20260806 10:48:01.057150 24109 tablet_bootstrap.cc:492] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Bootstrap starting.
I20260806 10:48:01.058822 24108 tablet_bootstrap.cc:654] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:01.059303 24109 tablet_bootstrap.cc:654] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:01.071100 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.057260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:52747)
0806 10:48:01.070558 (+ 13298us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.070749 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13128,"threads_started":1}
I20260806 10:48:01.072154 24014 heartbeater.cc:503] Master 127.16.21.60:41973 was elected leader, sending a full tablet report...
I20260806 10:48:01.074501 24108 tablet_bootstrap.cc:492] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: No bootstrap required, opened a new log
I20260806 10:48:01.074667 24108 ts_tablet_manager.cc:1397] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.013s
I20260806 10:48:01.075106 24109 tablet_bootstrap.cc:492] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: No bootstrap required, opened a new log
I20260806 10:48:01.075259 24109 ts_tablet_manager.cc:1397] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20260806 10:48:01.075567 24108 raft_consensus.cc:348] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.075749 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.074136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:51745)
0806 10:48:01.074343 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.074440 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:48:01.075785 24108 raft_consensus.cc:374] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:01.075897 24108 raft_consensus.cc:729] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f868ef8cfc45de8c64dcbb6c79d7ce, State: Initialized, Role: FOLLOWER
I20260806 10:48:01.076112 24109 raft_consensus.cc:348] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.076176 24108 consensus_queue.cc:260] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.076390 24109 raft_consensus.cc:374] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:01.076493 24109 raft_consensus.cc:729] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f96449697744f509e9262d4e1ce283b, State: Initialized, Role: FOLLOWER
I20260806 10:48:01.076869 24109 consensus_queue.cc:260] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.077466 24108 ts_tablet_manager.cc:1428] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:01.080538 23718 heartbeater.cc:503] Master 127.16.21.60:41973 was elected leader, sending a full tablet report...
I20260806 10:48:01.081851 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.078091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:40427)
0806 10:48:01.078778 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.078951 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":397,"threads_started":1}
I20260806 10:48:01.082815 24109 ts_tablet_manager.cc:1428] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 10:48:01.083154 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.082584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:54959)
0806 10:48:01.082931 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.083026 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260806 10:48:01.084837 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.083448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:41607)
0806 10:48:01.084211 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.084490 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":854,"threads_started":1}
I20260806 10:48:01.085073 23871 heartbeater.cc:503] Master 127.16.21.60:41973 was elected leader, sending a full tablet report...
I20260806 10:48:01.086365 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.085953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:48727)
0806 10:48:01.086110 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.086234 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 10:48:01.119917 24113 raft_consensus.cc:482] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:01.120272 24113 raft_consensus.cc:504] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.122449 24113 leader_election.cc:288] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205), 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:48:01.130088 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.123320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:48283 (local address 127.16.21.2:45963)
0806 10:48:01.123746 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.123752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.123782 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:48:01.123840 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.123844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.123898 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.124002 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:01.124575 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.125897 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.128139 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.129143 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.129292 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.129400 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.129707 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.129844 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260806 10:48:01.130137 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.122677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45963 (local address 127.16.21.1:48283)
0806 10:48:01.122984 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.123007 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:01.123183 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:01.124027 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:01.124032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:01.124059 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:01.124405 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.124419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.126043 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.126047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.127982 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.128000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.129350 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.129355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.129543 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.129567 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:48:01.129670 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 10:48:01.129808 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1}
I20260806 10:48:01.130800 23817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0170f2d5f0b4ca4b8649e2a76c2c614" candidate_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f96449697744f509e9262d4e1ce283b" is_pre_election: true
I20260806 10:48:01.131102 23817 raft_consensus.cc:2454] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f868ef8cfc45de8c64dcbb6c79d7ce in term 0.
I20260806 10:48:01.131645 23609 leader_election.cc:302] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f96449697744f509e9262d4e1ce283b, e7f868ef8cfc45de8c64dcbb6c79d7ce; no voters:
I20260806 10:48:01.132076 24113 raft_consensus.cc:2781] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:01.132210 24113 raft_consensus.cc:482] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:01.132345 24113 raft_consensus.cc:3037] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:01.133509 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.122762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41205 (local address 127.16.21.1:46013)
0806 10:48:01.123892 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.123920 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:01.124136 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:01.126085 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:01.126095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:01.126130 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:01.126719 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.126735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.128079 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.128087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.130717 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:01.130735 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.132568 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.132575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:01.132799 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.132842 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 10:48:01.132950 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 10:48:01.133076 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":74,"thread_start_us":787,"threads_started":1}
I20260806 10:48:01.133509 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.123098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:46013 (local address 127.16.21.3:41205)
0806 10:48:01.123549 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.123557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.124013 (+ 456us) server_negotiation.cc:407] Connection header received
0806 10:48:01.124338 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.124345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.124431 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.124519 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:01.126923 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.127925 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.130897 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.132122 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.132255 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.132395 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.133003 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.133179 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260806 10:48:01.134639 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0170f2d5f0b4ca4b8649e2a76c2c614" candidate_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc046a7def484b08a6b2e2ae73728e04" is_pre_election: true
I20260806 10:48:01.135119 23968 raft_consensus.cc:2454] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f868ef8cfc45de8c64dcbb6c79d7ce in term 0.
I20260806 10:48:01.135653 24113 raft_consensus.cc:504] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.136420 24113 leader_election.cc:288] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [CANDIDATE]: Term 1 election: Requested vote from peers fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205), 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963)
I20260806 10:48:01.136917 23817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0170f2d5f0b4ca4b8649e2a76c2c614" candidate_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f96449697744f509e9262d4e1ce283b"
I20260806 10:48:01.137100 23817 raft_consensus.cc:3037] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:01.139321 23817 raft_consensus.cc:2454] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f868ef8cfc45de8c64dcbb6c79d7ce in term 1.
I20260806 10:48:01.139714 23609 leader_election.cc:302] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f96449697744f509e9262d4e1ce283b, e7f868ef8cfc45de8c64dcbb6c79d7ce; no voters:
I20260806 10:48:01.139720 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0170f2d5f0b4ca4b8649e2a76c2c614" candidate_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc046a7def484b08a6b2e2ae73728e04"
I20260806 10:48:01.139900 23968 raft_consensus.cc:3037] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:01.142022 23968 raft_consensus.cc:2454] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f868ef8cfc45de8c64dcbb6c79d7ce in term 1.
I20260806 10:48:01.142307 24113 raft_consensus.cc:2781] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:01.142971 24113 raft_consensus.cc:686] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 LEADER]: Becoming Leader. State: Replica: e7f868ef8cfc45de8c64dcbb6c79d7ce, State: Running, Role: LEADER
I20260806 10:48:01.143365 24113 consensus_queue.cc:237] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } }
I20260806 10:48:01.144291 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.143847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:35315)
0806 10:48:01.143993 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.144104 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 10:48:01.147717 23538 catalog_manager.cc:5495] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1). New cstate: current_term: 1 leader_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f868ef8cfc45de8c64dcbb6c79d7ce" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34731 } health_report { overall_health: HEALTHY } } }
I20260806 10:48:01.178481 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.171714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.16.21.1:34731)
0806 10:48:01.171918 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.171922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.171939 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:48:01.172165 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.172168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.172246 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.172425 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:01.172949 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.173730 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.176079 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.176569 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.176663 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.177341 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:01.177907 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:01.178015 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.178170 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.178237 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 10:48:01.182602 23817 raft_consensus.cc:1261] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Refusing update from remote peer e7f868ef8cfc45de8c64dcbb6c79d7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:01.182803 23968 raft_consensus.cc:1261] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Refusing update from remote peer e7f868ef8cfc45de8c64dcbb6c79d7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:01.183420 24118 consensus_queue.cc:1035] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f96449697744f509e9262d4e1ce283b" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:01.183782 24113 consensus_queue.cc:1035] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc046a7def484b08a6b2e2ae73728e04" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:01.186591 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.185917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:37165)
0806 10:48:01.186221 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.186358 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260806 10:48:01.187997 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.187539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:39313)
0806 10:48:01.187694 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.187774 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 10:48:01.199568 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.198435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.2:49513)
0806 10:48:01.199229 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.199336 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635}
I20260806 10:48:01.202934 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.202476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:38231)
0806 10:48:01.202637 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.202743 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 10:48:01.205109 24122 mvcc.cc:205] Tried to move back new op lower bound from 7315510399716773888 to 7315510399567691776. Current Snapshot: MvccSnapshot[applied={T|T < 7315510399716773888 or (T in {7315510399716773888})}]
I20260806 10:48:01.213173 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.212663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.1:44897)
0806 10:48:01.212902 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.213010 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 10:48:01.221746 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.221158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.16.21.3:42945)
0806 10:48:01.221423 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.221526 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260806 10:48:01.224314 23538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:01.224594 23538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:01.235641 23538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7108e67ae9d0454b84f38a9095e42bb8: Sending DeleteTablet for 3 replicas of tablet c0170f2d5f0b4ca4b8649e2a76c2c614
I20260806 10:48:01.236729 23948 tablet_service.cc:1514] Processing DeleteTablet for tablet c0170f2d5f0b4ca4b8649e2a76c2c614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:01 UTC) from {username='oryx', principal='oryx/127.16.21.60@KRBTEST.COM'} at 127.0.0.1:60770
I20260806 10:48:01.238214 23653 tablet_service.cc:1514] Processing DeleteTablet for tablet c0170f2d5f0b4ca4b8649e2a76c2c614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:01 UTC) from {username='oryx', principal='oryx/127.16.21.60@KRBTEST.COM'} at 127.0.0.1:35460
I20260806 10:48:01.238241 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet c0170f2d5f0b4ca4b8649e2a76c2c614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:01 UTC) from {username='oryx', principal='oryx/127.16.21.60@KRBTEST.COM'} at 127.0.0.1:51888
I20260806 10:48:01.240309 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 23511
I20260806 10:48:01.251067 24132 tablet_replica.cc:330] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: stopping tablet replica
I20260806 10:48:01.251264 24132 raft_consensus.cc:2229] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:01.251578 24132 raft_consensus.cc:2258] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:01.253811 24132 ts_tablet_manager.cc:1905] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:01.260890 24133 tablet_replica.cc:330] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: stopping tablet replica
I20260806 10:48:01.261096 24133 raft_consensus.cc:2229] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:48:01.261794 24133 raft_consensus.cc:2258] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:01.262068 24131 tablet_replica.cc:330] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: stopping tablet replica
I20260806 10:48:01.262269 24131 raft_consensus.cc:2229] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:01.262570 24131 raft_consensus.cc:2258] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:01.262945 24133 ts_tablet_manager.cc:1905] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:01.265552 24131 ts_tablet_manager.cc:1905] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:01.270509 24132 ts_tablet_manager.cc:1918] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:01.271953 24131 ts_tablet_manager.cc:1918] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:01.272109 24131 log.cc:1196] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/c0170f2d5f0b4ca4b8649e2a76c2c614
I20260806 10:48:01.272321 24132 log.cc:1196] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/c0170f2d5f0b4ca4b8649e2a76c2c614
I20260806 10:48:01.272605 24131 ts_tablet_manager.cc:1939] T c0170f2d5f0b4ca4b8649e2a76c2c614 P fc046a7def484b08a6b2e2ae73728e04: Deleting consensus metadata
I20260806 10:48:01.272629 24133 ts_tablet_manager.cc:1918] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:01.272810 24133 log.cc:1196] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/c0170f2d5f0b4ca4b8649e2a76c2c614
I20260806 10:48:01.273442 24133 ts_tablet_manager.cc:1939] T c0170f2d5f0b4ca4b8649e2a76c2c614 P e7f868ef8cfc45de8c64dcbb6c79d7ce: Deleting consensus metadata
I20260806 10:48:01.273747 23524 catalog_manager.cc:4841] TS fc046a7def484b08a6b2e2ae73728e04 (127.16.21.3:41205): tablet c0170f2d5f0b4ca4b8649e2a76c2c614 (table test-table [id=fdda3e548ed34ee6b6be2756b74c785a]) successfully deleted
I20260806 10:48:01.273844 24132 ts_tablet_manager.cc:1939] T c0170f2d5f0b4ca4b8649e2a76c2c614 P 2f96449697744f509e9262d4e1ce283b: Deleting consensus metadata
I20260806 10:48:01.274581 23526 catalog_manager.cc:4841] TS e7f868ef8cfc45de8c64dcbb6c79d7ce (127.16.21.1:34731): tablet c0170f2d5f0b4ca4b8649e2a76c2c614 (table test-table [id=fdda3e548ed34ee6b6be2756b74c785a]) successfully deleted
I20260806 10:48:01.276162 23526 catalog_manager.cc:4841] TS 2f96449697744f509e9262d4e1ce283b (127.16.21.2:45963): tablet c0170f2d5f0b4ca4b8649e2a76c2c614 (table test-table [id=fdda3e548ed34ee6b6be2756b74c785a]) successfully deleted
I20260806 10:48:01.288872 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.241472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.16.21.1:34731)
0806 10:48:01.241624 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.241630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.261390 (+ 19760us) server_negotiation.cc:407] Connection header received
0806 10:48:01.261559 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.261566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.261699 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.261823 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:01.264589 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.265909 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.275706 (+ 9797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.276659 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.276867 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.280375 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:01.280393 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:01.280416 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:01.280490 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:01.285303 (+ 4813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:01.286118 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:01.286133 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:01.286139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:01.286415 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:01.286864 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:01.286872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:01.286878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:01.287403 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:01.287793 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.288038 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.288350 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:48:01.289589 23713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:01.410327 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 23511
I20260806 10:48:01.433655 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 23732
I20260806 10:48:01.447767 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.434416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.16.21.2:45963)
0806 10:48:01.434604 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.434610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.434709 (+ 99us) server_negotiation.cc:407] Connection header received
0806 10:48:01.434973 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.434977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.435041 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.435143 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:01.436807 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.437656 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.438900 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.439308 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.439490 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.441970 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:01.441986 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:01.442003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:01.442038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:01.445046 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:01.445728 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:01.445735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:01.445738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:01.445864 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:01.446244 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:01.446250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:01.446253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:01.446605 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:01.446909 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.447228 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.447453 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:48:01.448544 23865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:01.540102 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 23732
I20260806 10:48:01.561553 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 23884
I20260806 10:48:01.569677 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.568961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60102)
0806 10:48:01.569434 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.569515 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260806 10:48:01.571238 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.570773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45475 (local address 127.0.0.1:60116)
0806 10:48:01.570976 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:01.571077 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:45475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 10:48:01.575616 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:01.562356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.16.21.3:41205)
0806 10:48:01.562540 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:01.562549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:01.562609 (+ 60us) server_negotiation.cc:407] Connection header received
0806 10:48:01.562835 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:01.562839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:01.562902 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:01.563013 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:01.564443 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.565174 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.566415 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:01.566824 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:01.566960 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:01.569856 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:01.569873 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:01.569892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:01.569933 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:01.573229 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:01.573839 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:01.573846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:01.573849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:01.573989 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:01.574350 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:01.574355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:01.574357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:01.574708 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:01.574956 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:01.575198 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 10:48:01.575389 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:48:01.576251 24009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:01.698496 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 23884
I20260806 10:48:01.724912 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 23206
I20260806 10:48:01.725992 23383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:01.847513 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 23206
I20260806 10:48:01.872315 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 23311
I20260806 10:48:01.873371 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:01.996886 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 23311
2026-08-06T10:48:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9795 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:48:02 dist-test-slave-0b58 krb5kdc[24144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:48:02 dist-test-slave-0b58 krb5kdc[24144](info): set up 2 sockets
Aug 06 10:48:02 dist-test-slave-0b58 krb5kdc[24144](info): commencing operation
krb5kdc: starting...
W20260806 10:48:03.124935 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:48:03 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab.
Entry for principal oryx/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab.
I20260806 10:48:03.366062 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46471
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab
--principal=oryx/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:03.690555 24158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:03.690873 24158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:03.691047 24158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:03.698189 24158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:03.698328 24158 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 10:48:03.698431 24158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:03.698547 24158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:03.698642 24158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:03.698733 24158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:03.708999 24158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:46471
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.62.keytab
--principal=oryx/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:03.710265 24158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:03.711553 24158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:03.723656 24164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:03.723656 24166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:03.723690 24163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:03.724462 24158 server_base.cc:1034] running on GCE node
I20260806 10:48:03.725208 24158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:03.726472 24158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:03.727722 24158 hybrid_clock.cc:648] HybridClock initialized: now 1786013283727650 us; error 44 us; skew 500 ppm
I20260806 10:48:03.728004 24158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:03 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013283, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:03.731024 24158 init.cc:377] Logged in from keytab as oryx/127.16.21.62@KRBTEST.COM (short username oryx)
I20260806 10:48:03.732494 24158 webserver.cc:460] Webserver started at http://127.16.21.62:39059/ using document root <none> and password file <none>
I20260806 10:48:03.733155 24158 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:03.733268 24158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:03.733603 24158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:03.736606 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "d7d73d96537843af9b314195c83f4df0"
format_stamp: "Formatted at 2026-08-06 10:48:03 on dist-test-slave-0b58"
server_key: "ed7ecb5e2b180ded87bffe0d9c29fd3c"
server_key_iv: "b6259873b827105590b5fecacbffd07e"
server_key_version: "encryptionkey@0"
I20260806 10:48:03.737419 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "d7d73d96537843af9b314195c83f4df0"
format_stamp: "Formatted at 2026-08-06 10:48:03 on dist-test-slave-0b58"
server_key: "ed7ecb5e2b180ded87bffe0d9c29fd3c"
server_key_iv: "b6259873b827105590b5fecacbffd07e"
server_key_version: "encryptionkey@0"
I20260806 10:48:03.742329 24158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:48:03.745591 24173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:03.746876 24158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:48:03.747057 24158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "d7d73d96537843af9b314195c83f4df0"
format_stamp: "Formatted at 2026-08-06 10:48:03 on dist-test-slave-0b58"
server_key: "ed7ecb5e2b180ded87bffe0d9c29fd3c"
server_key_iv: "b6259873b827105590b5fecacbffd07e"
server_key_version: "encryptionkey@0"
I20260806 10:48:03.747269 24158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:03.780867 24158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:03.785055 24158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:03.785359 24158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:03.808713 24158 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:46471
I20260806 10:48:03.808738 24234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:46471 every 8 connection(s)
I20260806 10:48:03.810124 24158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:48:03.813930 24235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:03.818392 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 24158
I20260806 10:48:03.818617 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:48:03.819499 16468 external_mini_cluster.cc:1428] Setting key c754e174013227c7ad95d427b603d716
I20260806 10:48:03.826016 24235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Bootstrap starting.
I20260806 10:48:03.829941 24235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:03.831135 24235 log.cc:824] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:03.834626 24235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: No bootstrap required, opened a new log
Aug 06 10:48:03 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:03.844041 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:03.821693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.16.21.62:46471)
0806 10:48:03.822530 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:03.822542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:03.822586 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:48:03.824431 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:03.824453 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:03.824825 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:03.825137 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:03.827108 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:03.828871 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:03.830656 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:03.831400 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:03.831542 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:03.836768 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:03.836776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:03.836793 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:03.836832 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:03.839988 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:03.840828 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:03.840834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:03.840840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:03.841005 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:03.841395 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:03.841398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:03.841400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:03.841940 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:03.842192 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:03.842621 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 10:48:03.843006 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":403,"threads_started":1}
I20260806 10:48:03.844379 24235 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d73d96537843af9b314195c83f4df0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46471 } }
I20260806 10:48:03.844760 24235 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:03.844866 24235 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d73d96537843af9b314195c83f4df0, State: Initialized, Role: FOLLOWER
I20260806 10:48:03.845503 24235 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d73d96537843af9b314195c83f4df0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46471 } }
I20260806 10:48:03.845779 24235 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:48:03.845935 24235 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:48:03.846120 24235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:03.848685 24235 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d73d96537843af9b314195c83f4df0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46471 } }
I20260806 10:48:03.849161 24235 leader_election.cc:302] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7d73d96537843af9b314195c83f4df0; no voters:
I20260806 10:48:03.849784 24235 leader_election.cc:288] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:48:03.849948 24240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:03.851024 24240 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [term 1 LEADER]: Becoming Leader. State: Replica: d7d73d96537843af9b314195c83f4df0, State: Running, Role: LEADER
I20260806 10:48:03.851706 24240 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d73d96537843af9b314195c83f4df0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46471 } }
I20260806 10:48:03.851900 24235 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:48:03.855082 24242 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d73d96537843af9b314195c83f4df0. Latest consensus state: current_term: 1 leader_uuid: "d7d73d96537843af9b314195c83f4df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d73d96537843af9b314195c83f4df0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46471 } } }
I20260806 10:48:03.855654 24242 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:03.856060 24241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7d73d96537843af9b314195c83f4df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d73d96537843af9b314195c83f4df0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46471 } } }
I20260806 10:48:03.856383 24241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:03.859488 24248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:03.864195 24248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:03.872864 24248 catalog_manager.cc:1347] Generated new cluster ID: 69d3542611494d408e2d2ae8926ad907
I20260806 10:48:03.872982 24248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:03.881577 24248 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:48:03.882750 24248 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:03.891518 24248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Generated new TSK 0
I20260806 10:48:03.892230 24248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab.
Entry for principal oryx/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab.
I20260806 10:48:03.955363 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46471
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:04.289558 24261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:04.289829 24261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:04.289979 24261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:04.297037 24261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:04.297133 24261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:04.297305 24261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:48:04.297421 24261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:48:04.306794 24261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:04.307953 24261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:04.309253 24261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:04.321883 24267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:04.321918 24269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:04.322173 24266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:04.323122 24261 server_base.cc:1034] running on GCE node
I20260806 10:48:04.323680 24261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:04.324636 24261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:04.325879 24261 hybrid_clock.cc:648] HybridClock initialized: now 1786013284325820 us; error 35 us; skew 500 ppm
I20260806 10:48:04.326176 24261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:04 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013284, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:04.329389 24261 init.cc:377] Logged in from keytab as oryx/127.16.21.1@KRBTEST.COM (short username oryx)
I20260806 10:48:04.330760 24261 webserver.cc:460] Webserver started at http://127.16.21.1:44401/ using document root <none> and password file <none>
I20260806 10:48:04.331386 24261 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:04.331471 24261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:04.331737 24261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:04.334453 24261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "1ee5487a8ba645f29546931d691cf583"
format_stamp: "Formatted at 2026-08-06 10:48:04 on dist-test-slave-0b58"
server_key: "15bf4581aa02955495dd9056ade626d6"
server_key_iv: "63544c8b0b1dce6461e3d9e9495ec2c4"
server_key_version: "encryptionkey@0"
I20260806 10:48:04.335150 24261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "1ee5487a8ba645f29546931d691cf583"
format_stamp: "Formatted at 2026-08-06 10:48:04 on dist-test-slave-0b58"
server_key: "15bf4581aa02955495dd9056ade626d6"
server_key_iv: "63544c8b0b1dce6461e3d9e9495ec2c4"
server_key_version: "encryptionkey@0"
I20260806 10:48:04.340160 24261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:48:04.343371 24276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:04.344648 24261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:04.344775 24261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "1ee5487a8ba645f29546931d691cf583"
format_stamp: "Formatted at 2026-08-06 10:48:04 on dist-test-slave-0b58"
server_key: "15bf4581aa02955495dd9056ade626d6"
server_key_iv: "63544c8b0b1dce6461e3d9e9495ec2c4"
server_key_version: "encryptionkey@0"
I20260806 10:48:04.344972 24261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:04.357442 24261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:04.362491 24261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:04.362762 24261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:04.365340 24261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:04.365442 24261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:04.365586 24261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:04.365665 24261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:48:04 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013284, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:04.390508 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.365944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:59087 (local address 127.16.21.62:46471)
0806 10:48:04.366578 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:04.366583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:04.367654 (+ 1071us) server_negotiation.cc:407] Connection header received
0806 10:48:04.369874 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:04.369880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:04.369968 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:04.370106 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:04.372645 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.373575 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.376402 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.377082 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.377215 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.382323 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:04.382329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:04.382337 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:04.382383 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:04.386638 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.387552 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.387561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.387567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.387732 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.388357 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.388365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.388373 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.388808 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:04.388986 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:04.389891 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 10:48:04.390115 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":420,"threads_started":1}
I20260806 10:48:04.392041 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.366666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.16.21.1:59087)
0806 10:48:04.367500 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:04.367536 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:04.369594 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:04.370378 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:04.370390 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:04.371365 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:04.372402 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.372451 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.373817 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.373824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.376198 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.376211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.377570 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.377581 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.377927 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.378704 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:04.378743 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:04.381997 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:04.386799 (+ 4802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.386806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.386816 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.387381 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.387877 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.387887 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.387893 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.388191 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.388984 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:04.388993 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:04.389499 (+ 506us) client_negotiation.cc:769] Sending connection context
0806 10:48:04.389801 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 10:48:04.390325 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":366,"threads_started":1}
I20260806 10:48:04.411221 24261 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:37405
I20260806 10:48:04.411238 24397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:37405 every 8 connection(s)
I20260806 10:48:04.412824 24261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:48:04.414880 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.399323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:51863 (local address 127.16.21.62:46471)
0806 10:48:04.399708 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:04.399714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:04.400101 (+ 387us) server_negotiation.cc:407] Connection header received
0806 10:48:04.400362 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:04.400368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:04.400439 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:04.400537 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:04.401930 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.402633 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.404745 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.405533 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.405699 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.408389 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:04.408395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:04.408401 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:04.408457 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:04.411973 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.412673 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.412678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.412683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.412857 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.413337 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.413345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.413349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.413716 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:04.414123 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:04.414476 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 10:48:04.414667 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260806 10:48:04.414860 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.399340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.16.21.1:51863)
0806 10:48:04.399979 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:04.400003 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:04.400161 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:04.400565 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:04.400570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:04.401253 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:04.401770 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.401788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.402768 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.402781 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.404563 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.404582 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.405768 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.405779 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.405990 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.406713 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:04.406748 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:04.408112 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:04.412102 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.412106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.412108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.412557 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.413004 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.413008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.413011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.413171 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.413869 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:04.413873 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:04.414123 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 10:48:04.414341 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 10:48:04.414572 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":380,"threads_started":1}
I20260806 10:48:04.415917 24398 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46471
I20260806 10:48:04.416144 24398 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:04.416960 24398 heartbeater.cc:511] Master 127.16.21.62:46471 requested a full tablet report, sending...
I20260806 10:48:04.418846 24190 ts_manager.cc:194] Registered new tserver with Master: 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405)
I20260806 10:48:04.420019 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 24261
I20260806 10:48:04.420223 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:48:04.420360 24190 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:59087
I20260806 10:48:04.420977 16468 external_mini_cluster.cc:1428] Setting key 3f956fab8028bf7ebff7ba7c87cc0cfc
WARNING: no policy specified for oryx/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab.
Entry for principal oryx/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab.
I20260806 10:48:04.478843 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46471
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:04.812587 24403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:04.812881 24403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:04.813050 24403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:04.820334 24403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:04.820472 24403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:04.820611 24403 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:48:04.820755 24403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:48:04.830464 24403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:04.831708 24403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:04.833114 24403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:04.846184 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:04.846113 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:04.846220 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:04.847476 24403 server_base.cc:1034] running on GCE node
I20260806 10:48:04.848165 24403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:04.849164 24403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:04.850401 24403 hybrid_clock.cc:648] HybridClock initialized: now 1786013284850324 us; error 53 us; skew 500 ppm
I20260806 10:48:04.850688 24403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:04 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013284, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:04.853909 24403 init.cc:377] Logged in from keytab as oryx/127.16.21.2@KRBTEST.COM (short username oryx)
I20260806 10:48:04.855253 24403 webserver.cc:460] Webserver started at http://127.16.21.2:38165/ using document root <none> and password file <none>
I20260806 10:48:04.855886 24403 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:04.855971 24403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:04.856236 24403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:04.859128 24403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "3287a7af1e9943aea787d18b0c885510"
format_stamp: "Formatted at 2026-08-06 10:48:04 on dist-test-slave-0b58"
server_key: "f6b800487223c8fb15bc90e0bc624107"
server_key_iv: "1664ac8c4d65f2e56858ffa25f38caf8"
server_key_version: "encryptionkey@0"
I20260806 10:48:04.859872 24403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "3287a7af1e9943aea787d18b0c885510"
format_stamp: "Formatted at 2026-08-06 10:48:04 on dist-test-slave-0b58"
server_key: "f6b800487223c8fb15bc90e0bc624107"
server_key_iv: "1664ac8c4d65f2e56858ffa25f38caf8"
server_key_version: "encryptionkey@0"
I20260806 10:48:04.864912 24403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 10:48:04.868188 24419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:04.869485 24403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:48:04.869642 24403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "3287a7af1e9943aea787d18b0c885510"
format_stamp: "Formatted at 2026-08-06 10:48:04 on dist-test-slave-0b58"
server_key: "f6b800487223c8fb15bc90e0bc624107"
server_key_iv: "1664ac8c4d65f2e56858ffa25f38caf8"
server_key_version: "encryptionkey@0"
I20260806 10:48:04.869868 24403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:04.897292 24403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:04.901723 24403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:04.902014 24403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:04.904681 24403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:04.904799 24403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:04.904970 24403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:04.905047 24403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:48:04 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013284, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:04.929435 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.905444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:38143 (local address 127.16.21.62:46471)
0806 10:48:04.905655 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:04.905661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:04.907100 (+ 1439us) server_negotiation.cc:407] Connection header received
0806 10:48:04.909353 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:04.909364 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:04.909481 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:04.909609 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:04.911843 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.912923 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.915524 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.915975 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.916117 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.921159 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:04.921164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:04.921168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:04.921198 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:04.925111 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.926094 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.926103 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.926109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.926319 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.927028 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.927037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.927042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.927492 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:04.927675 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:04.928590 (+ 915us) server_negotiation.cc:299] Negotiation successful
0806 10:48:04.928900 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:48:04.929982 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.905938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.16.21.2:38143)
0806 10:48:04.906904 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:04.906954 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:04.908957 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:04.909881 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:04.909889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:04.910674 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:04.911565 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.911599 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.913091 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.913095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.915331 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.915344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.916239 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.916247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.916503 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.917357 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:04.917397 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:04.920928 (+ 3531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:04.925308 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.925320 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.925340 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.925955 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.926488 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.926496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.926507 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.926856 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.927681 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:04.927691 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:04.928092 (+ 401us) client_negotiation.cc:769] Sending connection context
0806 10:48:04.928477 (+ 385us) client_negotiation.cc:240] Negotiation successful
0806 10:48:04.928906 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":446,"threads_started":1}
I20260806 10:48:04.951699 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.936235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.16.21.2:35397)
0806 10:48:04.936998 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:04.937013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:04.937168 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:04.937503 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:04.937509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:04.937974 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:04.938265 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.938274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.939597 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.939602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.941362 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:04.941386 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.942563 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.942573 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:04.942798 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.943555 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:04.943585 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:04.944631 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:04.948242 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.948247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.948251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.948988 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.949453 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:04.949459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:04.949468 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:04.949651 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:04.950556 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:04.950559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:04.950740 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 10:48:04.950932 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 10:48:04.951285 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":482,"threads_started":1}
I20260806 10:48:04.952499 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:04.936133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:35397 (local address 127.16.21.62:46471)
0806 10:48:04.936368 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:04.936375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:04.937099 (+ 724us) server_negotiation.cc:407] Connection header received
0806 10:48:04.937299 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:04.937304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:04.937363 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:04.937463 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:04.938404 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.939446 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.941574 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:04.942181 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:04.942308 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:04.944816 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:04.944823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:04.944827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:04.944865 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:04.948108 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.949132 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.949143 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.949145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.949326 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:04.949830 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:04.949834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:04.949836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:04.950349 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:04.950501 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:04.952011 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0806 10:48:04.952220 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:48:04.959296 24403 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:46079
I20260806 10:48:04.959326 24540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:46079 every 8 connection(s)
I20260806 10:48:04.960526 24403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:48:04.962507 24541 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46471
I20260806 10:48:04.962736 24541 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:04.963349 24541 heartbeater.cc:511] Master 127.16.21.62:46471 requested a full tablet report, sending...
I20260806 10:48:04.964126 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 24403
I20260806 10:48:04.964340 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:48:04.964856 16468 external_mini_cluster.cc:1428] Setting key dc922a625809e2d13f96baca96486b2d
I20260806 10:48:04.964879 24190 ts_manager.cc:194] Registered new tserver with Master: 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:04.965816 24190 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:38143
WARNING: no policy specified for oryx/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab.
Entry for principal oryx/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab.
I20260806 10:48:05.021301 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46471
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:05.356690 24545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:05.357064 24545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:05.357261 24545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:05.367055 24545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:05.367190 24545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:05.367345 24545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:48:05.367494 24545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:48:05.377240 24545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.21.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:05.378316 24545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:05.379515 24545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:05.392333 24553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:05.392437 24550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:05.392441 24551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:05.393425 24545 server_base.cc:1034] running on GCE node
I20260806 10:48:05.394019 24545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:05.395023 24545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:05.396306 24545 hybrid_clock.cc:648] HybridClock initialized: now 1786013285396183 us; error 45 us; skew 500 ppm
I20260806 10:48:05.396598 24545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:05 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013285, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:05.399570 24545 init.cc:377] Logged in from keytab as oryx/127.16.21.3@KRBTEST.COM (short username oryx)
I20260806 10:48:05.400965 24545 webserver.cc:460] Webserver started at http://127.16.21.3:36291/ using document root <none> and password file <none>
I20260806 10:48:05.401610 24545 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:05.401762 24545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:05.402091 24545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:05.404953 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "500ad2656f9b4f4791da216a46876e56"
format_stamp: "Formatted at 2026-08-06 10:48:05 on dist-test-slave-0b58"
server_key: "617a971d525bfaaf5c5ccca6038298fc"
server_key_iv: "54fd720777dd459ce96d842acbf2b23c"
server_key_version: "encryptionkey@0"
I20260806 10:48:05.405696 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "500ad2656f9b4f4791da216a46876e56"
format_stamp: "Formatted at 2026-08-06 10:48:05 on dist-test-slave-0b58"
server_key: "617a971d525bfaaf5c5ccca6038298fc"
server_key_iv: "54fd720777dd459ce96d842acbf2b23c"
server_key_version: "encryptionkey@0"
I20260806 10:48:05.410843 24545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 10:48:05.415155 24560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:05.416700 24545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:05.416833 24545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "500ad2656f9b4f4791da216a46876e56"
format_stamp: "Formatted at 2026-08-06 10:48:05 on dist-test-slave-0b58"
server_key: "617a971d525bfaaf5c5ccca6038298fc"
server_key_iv: "54fd720777dd459ce96d842acbf2b23c"
server_key_version: "encryptionkey@0"
I20260806 10:48:05.417016 24545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:05.423307 24398 heartbeater.cc:503] Master 127.16.21.62:46471 was elected leader, sending a full tablet report...
I20260806 10:48:05.436313 24545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:05.439894 24545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:05.440102 24545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:05.442559 24545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:05.442639 24545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:05.442818 24545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:05.442895 24545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:48:05 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013285, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:05.469264 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.443325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:33473 (local address 127.16.21.62:46471)
0806 10:48:05.443519 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.443525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.445083 (+ 1558us) server_negotiation.cc:407] Connection header received
0806 10:48:05.447236 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.447242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.447335 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.447459 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:05.450011 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.451204 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.453661 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.454443 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.454572 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.461045 (+ 6473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:05.461050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:05.461053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:05.461084 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:05.464848 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:05.465930 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:05.465935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:05.465940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:05.466084 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:05.466599 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:05.466604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:05.466608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:05.467012 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:05.467191 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.468403 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.468703 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:48:05.469869 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.443995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.16.21.3:33473)
0806 10:48:05.444912 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.444970 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.446982 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.447735 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.447748 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.448772 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:05.449759 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.449785 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.451396 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.451405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.453439 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.453460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.454669 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.454676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.454915 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.455715 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:05.455773 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:05.460802 (+ 5029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:05.464999 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:05.465008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:05.465027 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:05.465674 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:05.466237 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:05.466242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:05.466246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:05.466468 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:05.467159 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:05.467168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:05.467716 (+ 548us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.468078 (+ 362us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.468703 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":407,"threads_started":1}
I20260806 10:48:05.490483 24545 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:36481
I20260806 10:48:05.490525 24681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:36481 every 8 connection(s)
I20260806 10:48:05.491710 24545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:48:05.491940 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.477311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:55199 (local address 127.16.21.62:46471)
0806 10:48:05.477514 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.477521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.478670 (+ 1149us) server_negotiation.cc:407] Connection header received
0806 10:48:05.478879 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.478886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.478957 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.479055 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:05.480103 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.481131 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.482581 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.483211 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.483367 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.485933 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:05.485941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:05.485948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:05.485994 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:05.489193 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:05.489910 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:05.489916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:05.489920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:05.490047 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:05.490409 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:05.490416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:05.490420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:05.490771 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:05.490920 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.491326 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.491563 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:48:05.492086 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.477778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.16.21.3:55199)
0806 10:48:05.478557 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.478583 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.478738 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.479087 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.479092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.479596 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:05.479948 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.479958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.481289 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.481294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.482419 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.482431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.483626 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.483631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.483780 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.484671 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:05.484690 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:05.485735 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:05.489348 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:05.489353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:05.489357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:05.489775 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:05.490160 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:05.490165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:05.490168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:05.490297 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:05.490886 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:05.490891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:05.491074 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.491225 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.491571 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":459,"threads_started":1}
I20260806 10:48:05.494143 24682 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46471
I20260806 10:48:05.494372 24682 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:05.495054 24682 heartbeater.cc:511] Master 127.16.21.62:46471 requested a full tablet report, sending...
I20260806 10:48:05.496417 24189 ts_manager.cc:194] Registered new tserver with Master: 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481)
I20260806 10:48:05.496606 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 24545
I20260806 10:48:05.496765 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:48:05.497207 16468 external_mini_cluster.cc:1428] Setting key 4b50bd377871d0857676e68c29a8b2d6
I20260806 10:48:05.497380 24189 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:33473
I20260806 10:48:05.500294 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:48:05 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:48:05 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:05 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:48:05.560847 24285 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:05.550070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.16.21.62:46471)
0806 10:48:05.550311 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.550320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.550549 (+ 229us) server_negotiation.cc:407] Connection header received
0806 10:48:05.550769 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.550779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.550845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.550953 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:05.552420 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.553183 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.554432 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.554845 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.554985 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.560522 (+ 5537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52138: BlockingRecv error: recv got EOF from 127.0.0.1:52138 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 10:48:05 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:05.577656 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.563937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.16.21.62:46471)
0806 10:48:05.564141 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.564149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.564535 (+ 386us) server_negotiation.cc:407] Connection header received
0806 10:48:05.564717 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.564722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.564794 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.564895 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:05.565986 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.566712 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.567910 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.568471 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.568655 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.572300 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:05.572308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:05.572316 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:05.572360 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:05.574992 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:05.575623 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:05.575633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:05.575635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:05.575752 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:05.576132 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:05.576137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:05.576140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:05.576518 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:05.576840 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.577079 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.577310 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:48:05.581583 24189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:48:05.588590 24189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:05.611184 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.599986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:36481 (local address 127.0.0.1:41866)
0806 10:48:05.601229 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.601283 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.601480 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.603471 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.603482 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.603539 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:05.604144 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.604162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.606591 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.606596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.608711 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.608723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.610025 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.610032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.610379 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.610456 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.610585 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.610784 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":124,"thread_start_us":780,"threads_started":1}
I20260806 10:48:05.611213 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.600447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.16.21.3:36481)
0806 10:48:05.602707 (+ 2260us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.602719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.602755 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:48:05.602851 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.602858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.603153 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.603724 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:05.604920 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.606448 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.608878 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.609813 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.609956 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.610215 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.610666 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.610869 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2053,"thread_start_us":1872,"threads_started":1}
I20260806 10:48:05.613318 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.601110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.16.21.1:37405)
0806 10:48:05.602107 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.602116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.604215 (+ 2099us) server_negotiation.cc:407] Connection header received
0806 10:48:05.604479 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.604487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.604712 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.604892 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:05.606009 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.607532 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.610497 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.612378 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.612506 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.612708 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.612981 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.613095 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":1118,"threads_started":1}
I20260806 10:48:05.613453 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.600631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37405 (local address 127.0.0.1:46832)
0806 10:48:05.604080 (+ 3449us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.604106 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.604236 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.605166 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.605172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.605229 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:05.605826 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.605839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.607690 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.607704 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.610343 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.610352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.612601 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.612607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.612783 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.612820 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.612925 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.613051 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2803,"mutex_wait_us":156,"spinlock_wait_cycles":5632,"thread_start_us":2975,"threads_started":1}
I20260806 10:48:05.615044 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 1908971bb3b3423eaed78d1902259cb8 (DEFAULT_TABLE table=test-table [id=f2f58e2301a4465da3eb282af72e58aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:05.616284 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1908971bb3b3423eaed78d1902259cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:05.618080 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.601512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46079 (local address 127.0.0.1:45104)
0806 10:48:05.604579 (+ 3067us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.604601 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.604714 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.605162 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.605166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.605191 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:05.605704 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.605713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.607646 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.607651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.610314 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.610334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.617234 (+ 6900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.617243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.617430 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.617468 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.617570 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.617698 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2963,"mutex_wait_us":119,"thread_start_us":1773,"threads_started":1}
I20260806 10:48:05.618103 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.602049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.16.21.2:46079)
0806 10:48:05.603090 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.603102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.604695 (+ 1593us) server_negotiation.cc:407] Connection header received
0806 10:48:05.604831 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.604836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.604989 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.605168 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:05.605865 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.607464 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.610515 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.616749 (+ 6234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.616893 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.617088 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.617642 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.617826 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":662,"threads_started":1}
I20260806 10:48:05.619015 24328 tablet_service.cc:1467] Processing CreateTablet for tablet 1908971bb3b3423eaed78d1902259cb8 (DEFAULT_TABLE table=test-table [id=f2f58e2301a4465da3eb282af72e58aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:05.620330 24328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1908971bb3b3423eaed78d1902259cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:05.621265 24469 tablet_service.cc:1467] Processing CreateTablet for tablet 1908971bb3b3423eaed78d1902259cb8 (DEFAULT_TABLE table=test-table [id=f2f58e2301a4465da3eb282af72e58aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:05.622577 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1908971bb3b3423eaed78d1902259cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:05.632529 24707 tablet_bootstrap.cc:492] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Bootstrap starting.
I20260806 10:48:05.634616 24708 tablet_bootstrap.cc:492] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Bootstrap starting.
I20260806 10:48:05.636495 24707 tablet_bootstrap.cc:654] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:05.636947 24709 tablet_bootstrap.cc:492] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Bootstrap starting.
I20260806 10:48:05.637997 24707 log.cc:824] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:05.638064 24708 tablet_bootstrap.cc:654] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:05.639159 24708 log.cc:824] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:05.640223 24709 tablet_bootstrap.cc:654] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:05.641289 24709 log.cc:824] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:05.641955 24707 tablet_bootstrap.cc:492] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: No bootstrap required, opened a new log
I20260806 10:48:05.642175 24708 tablet_bootstrap.cc:492] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: No bootstrap required, opened a new log
I20260806 10:48:05.642243 24707 ts_tablet_manager.cc:1397] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 10:48:05.642439 24708 ts_tablet_manager.cc:1397] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:48:05.647439 24709 tablet_bootstrap.cc:492] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: No bootstrap required, opened a new log
I20260806 10:48:05.647691 24709 ts_tablet_manager.cc:1397] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 10:48:05.653100 24707 raft_consensus.cc:348] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.653537 24707 raft_consensus.cc:374] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:05.653654 24707 raft_consensus.cc:729] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Initialized, Role: FOLLOWER
I20260806 10:48:05.653863 24708 raft_consensus.cc:348] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.654270 24708 raft_consensus.cc:374] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:05.654394 24708 raft_consensus.cc:729] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Initialized, Role: FOLLOWER
I20260806 10:48:05.654299 24707 consensus_queue.cc:260] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.655992 24707 ts_tablet_manager.cc:1428] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 10:48:05.655004 24708 consensus_queue.cc:260] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.656548 24709 raft_consensus.cc:348] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.656962 24709 raft_consensus.cc:374] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:05.657066 24709 raft_consensus.cc:729] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3287a7af1e9943aea787d18b0c885510, State: Initialized, Role: FOLLOWER
I20260806 10:48:05.657579 24709 consensus_queue.cc:260] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.657828 24708 ts_tablet_manager.cc:1428] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260806 10:48:05.658361 24682 heartbeater.cc:503] Master 127.16.21.62:46471 was elected leader, sending a full tablet report...
I20260806 10:48:05.658982 24541 heartbeater.cc:503] Master 127.16.21.62:46471 was elected leader, sending a full tablet report...
I20260806 10:48:05.659498 24709 ts_tablet_manager.cc:1428] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260806 10:48:05.665377 24399 tablet.cc:2369] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:48:05.712635 24542 tablet.cc:2369] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:48:05.743458 24683 tablet.cc:2369] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:48:05.860947 24713 raft_consensus.cc:482] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:05.861232 24713 raft_consensus.cc:504] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.862886 24713 leader_election.cc:288] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481), 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:05.870173 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.863348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:38181 (local address 127.16.21.3:36481)
0806 10:48:05.863550 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.863554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.864143 (+ 589us) server_negotiation.cc:407] Connection header received
0806 10:48:05.864237 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.864242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.864348 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.864438 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:05.865255 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.866231 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.868323 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.869169 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.869267 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.869351 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.869765 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.869897 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:48:05.870219 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.864171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:40111 (local address 127.16.21.2:46079)
0806 10:48:05.864422 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.864427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.864692 (+ 265us) server_negotiation.cc:407] Connection header received
0806 10:48:05.865000 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.865006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.865089 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.865190 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:05.865810 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.866731 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.868516 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.869312 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.869453 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.869559 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.869833 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.869968 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 10:48:05.870316 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.863121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:36481 (local address 127.16.21.1:38181)
0806 10:48:05.863796 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.863815 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.863972 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.864633 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.864640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.864674 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:05.865097 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.865112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.866368 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.866374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.868127 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.868140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.869380 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.869390 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.869595 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.869632 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.869705 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.869847 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":54,"thread_start_us":326,"threads_started":1}
I20260806 10:48:05.870530 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.863889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46079 (local address 127.16.21.1:40111)
0806 10:48:05.864556 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:05.864597 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:05.864838 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:05.865234 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:05.865238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:05.865263 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:05.865653 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.865666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.866865 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.866868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.868362 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:05.868378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.869534 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.869538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:05.869648 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.869671 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:48:05.869793 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 10:48:05.869921 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":3,"thread_start_us":285,"threads_started":1}
I20260806 10:48:05.871420 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1908971bb3b3423eaed78d1902259cb8" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ad2656f9b4f4791da216a46876e56" is_pre_election: true
I20260806 10:48:05.871537 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1908971bb3b3423eaed78d1902259cb8" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510" is_pre_election: true
I20260806 10:48:05.871984 24489 raft_consensus.cc:2454] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 0.
I20260806 10:48:05.871990 24630 raft_consensus.cc:2454] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 0.
I20260806 10:48:05.872682 24280 leader_election.cc:302] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee5487a8ba645f29546931d691cf583, 3287a7af1e9943aea787d18b0c885510; no voters:
I20260806 10:48:05.873068 24713 raft_consensus.cc:2781] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:05.873225 24713 raft_consensus.cc:482] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:05.873332 24713 raft_consensus.cc:3037] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:05.875576 24713 raft_consensus.cc:504] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.876071 24713 leader_election.cc:288] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 election: Requested vote from peers 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481), 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:05.876484 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1908971bb3b3423eaed78d1902259cb8" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ad2656f9b4f4791da216a46876e56"
I20260806 10:48:05.876608 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1908971bb3b3423eaed78d1902259cb8" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510"
I20260806 10:48:05.876664 24630 raft_consensus.cc:3037] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:05.876820 24489 raft_consensus.cc:3037] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:05.878849 24630 raft_consensus.cc:2454] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 1.
I20260806 10:48:05.878988 24489 raft_consensus.cc:2454] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 1.
I20260806 10:48:05.879252 24280 leader_election.cc:302] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee5487a8ba645f29546931d691cf583, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:05.879510 24713 raft_consensus.cc:2781] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:05.879956 24713 raft_consensus.cc:686] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Running, Role: LEADER
I20260806 10:48:05.880494 24713 consensus_queue.cc:237] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:05.884789 24188 catalog_manager.cc:5495] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ee5487a8ba645f29546931d691cf583 (127.16.21.1). New cstate: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } health_report { overall_health: HEALTHY } } }
I20260806 10:48:05.903558 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:05.898204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.16.21.1:37405)
0806 10:48:05.898412 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:05.898420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:05.898543 (+ 123us) server_negotiation.cc:407] Connection header received
0806 10:48:05.898777 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:05.898783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:05.898885 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:05.899033 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:05.899425 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.900156 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.901473 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:05.901892 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:05.902033 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:05.902471 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:05.902974 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:05.903122 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:05.903209 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 10:48:05.903289 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:48:05.911140 24630 raft_consensus.cc:1261] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Refusing update from remote peer 1ee5487a8ba645f29546931d691cf583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:05.911239 24489 raft_consensus.cc:1261] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Refusing update from remote peer 1ee5487a8ba645f29546931d691cf583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:05.912132 24718 consensus_queue.cc:1035] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:05.912559 24713 consensus_queue.cc:1035] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:05.925328 24721 mvcc.cc:205] Tried to move back new op lower bound from 7315510419083153408 to 7315510418970267648. Current Snapshot: MvccSnapshot[applied={T|T < 7315510419083153408}]
I20260806 10:48:05.926985 24722 mvcc.cc:205] Tried to move back new op lower bound from 7315510419083153408 to 7315510418970267648. Current Snapshot: MvccSnapshot[applied={T|T < 7315510419083153408}]
I20260806 10:48:05.930012 24723 mvcc.cc:205] Tried to move back new op lower bound from 7315510419083153408 to 7315510418970267648. Current Snapshot: MvccSnapshot[applied={T|T < 7315510419083153408 or (T in {7315510419083153408})}]
I20260806 10:48:05.939937 24188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:05.940222 24188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:05.943809 24188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Sending DeleteTablet for 3 replicas of tablet 1908971bb3b3423eaed78d1902259cb8
I20260806 10:48:05.944883 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1908971bb3b3423eaed78d1902259cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:05 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41866
I20260806 10:48:05.945035 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet 1908971bb3b3423eaed78d1902259cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:05 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260806 10:48:05.945155 24328 tablet_service.cc:1514] Processing DeleteTablet for tablet 1908971bb3b3423eaed78d1902259cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:05 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:46832
I20260806 10:48:05.949055 24188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:48:05.950244 24188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:05.959547 24731 tablet_replica.cc:330] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: stopping tablet replica
I20260806 10:48:05.961215 24327 tablet_service.cc:1467] Processing CreateTablet for tablet 18a815a3ee804ec784ae59deb672f261 (DEFAULT_TABLE table=test-table [id=03a7926e7c7b40beaddfd8f70d1a8ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:05.961203 24468 tablet_service.cc:1467] Processing CreateTablet for tablet 18a815a3ee804ec784ae59deb672f261 (DEFAULT_TABLE table=test-table [id=03a7926e7c7b40beaddfd8f70d1a8ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:05.961812 24327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a815a3ee804ec784ae59deb672f261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:05.961810 24468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a815a3ee804ec784ae59deb672f261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:05.969316 24707 tablet_bootstrap.cc:492] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Bootstrap starting.
I20260806 10:48:05.971316 24707 tablet_bootstrap.cc:654] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:05.974231 24709 tablet_bootstrap.cc:492] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Bootstrap starting.
I20260806 10:48:05.975802 24709 tablet_bootstrap.cc:654] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:05.965503 24609 tablet_service.cc:1467] Processing CreateTablet for tablet 18a815a3ee804ec784ae59deb672f261 (DEFAULT_TABLE table=test-table [id=03a7926e7c7b40beaddfd8f70d1a8ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:05.977566 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a815a3ee804ec784ae59deb672f261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:05.983357 24707 tablet_bootstrap.cc:492] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: No bootstrap required, opened a new log
I20260806 10:48:05.983520 24707 ts_tablet_manager.cc:1397] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20260806 10:48:05.984495 24707 raft_consensus.cc:348] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:05.984763 24707 raft_consensus.cc:374] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:05.984884 24707 raft_consensus.cc:729] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Initialized, Role: FOLLOWER
I20260806 10:48:05.985149 24707 consensus_queue.cc:260] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:05.985773 24707 ts_tablet_manager.cc:1428] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260806 10:48:05.987774 24708 tablet_bootstrap.cc:492] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Bootstrap starting.
I20260806 10:48:05.988883 24733 tablet_replica.cc:330] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: stopping tablet replica
I20260806 10:48:05.989439 24733 raft_consensus.cc:2229] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:48:05.989548 24708 tablet_bootstrap.cc:654] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:05.990186 24733 raft_consensus.cc:2258] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:05.990952 24731 raft_consensus.cc:2229] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:05.991470 24731 raft_consensus.cc:2258] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:05.992399 24732 tablet_replica.cc:330] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: stopping tablet replica
I20260806 10:48:05.992940 24733 ts_tablet_manager.cc:1905] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:05.992906 24732 raft_consensus.cc:2229] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:05.993480 24732 raft_consensus.cc:2258] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:05.993762 24731 ts_tablet_manager.cc:1905] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:05.996424 24708 tablet_bootstrap.cc:492] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: No bootstrap required, opened a new log
I20260806 10:48:05.996604 24708 ts_tablet_manager.cc:1397] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260806 10:48:05.997432 24708 raft_consensus.cc:348] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:05.997702 24708 raft_consensus.cc:374] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:05.997845 24708 raft_consensus.cc:729] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Initialized, Role: FOLLOWER
I20260806 10:48:05.998202 24708 consensus_queue.cc:260] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.001726 24733 ts_tablet_manager.cc:1918] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:06.001940 24733 log.cc:1196] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/1908971bb3b3423eaed78d1902259cb8
I20260806 10:48:06.002218 24708 ts_tablet_manager.cc:1428] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 10:48:06.002626 24733 ts_tablet_manager.cc:1939] T 1908971bb3b3423eaed78d1902259cb8 P 1ee5487a8ba645f29546931d691cf583: Deleting consensus metadata
I20260806 10:48:06.004423 24174 catalog_manager.cc:4841] TS 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405): tablet 1908971bb3b3423eaed78d1902259cb8 (table test-table [id=f2f58e2301a4465da3eb282af72e58aa]) successfully deleted
I20260806 10:48:06.005340 24732 ts_tablet_manager.cc:1905] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:06.006824 24731 ts_tablet_manager.cc:1918] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:06.006994 24731 log.cc:1196] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/1908971bb3b3423eaed78d1902259cb8
I20260806 10:48:06.007508 24731 ts_tablet_manager.cc:1939] T 1908971bb3b3423eaed78d1902259cb8 P 500ad2656f9b4f4791da216a46876e56: Deleting consensus metadata
I20260806 10:48:06.009261 24177 catalog_manager.cc:4841] TS 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481): tablet 1908971bb3b3423eaed78d1902259cb8 (table test-table [id=f2f58e2301a4465da3eb282af72e58aa]) successfully deleted
I20260806 10:48:06.010288 24709 tablet_bootstrap.cc:492] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: No bootstrap required, opened a new log
I20260806 10:48:06.010457 24709 ts_tablet_manager.cc:1397] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.005s
I20260806 10:48:06.011274 24709 raft_consensus.cc:348] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.011437 24732 ts_tablet_manager.cc:1918] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:06.011513 24709 raft_consensus.cc:374] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.011629 24732 log.cc:1196] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/1908971bb3b3423eaed78d1902259cb8
I20260806 10:48:06.011646 24709 raft_consensus.cc:729] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3287a7af1e9943aea787d18b0c885510, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.012003 24709 consensus_queue.cc:260] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.012281 24732 ts_tablet_manager.cc:1939] T 1908971bb3b3423eaed78d1902259cb8 P 3287a7af1e9943aea787d18b0c885510: Deleting consensus metadata
I20260806 10:48:06.013166 24709 ts_tablet_manager.cc:1428] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 10:48:06.014006 24177 catalog_manager.cc:4841] TS 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079): tablet 1908971bb3b3423eaed78d1902259cb8 (table test-table [id=f2f58e2301a4465da3eb282af72e58aa]) successfully deleted
I20260806 10:48:06.063488 24715 raft_consensus.cc:482] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:06.063761 24715 raft_consensus.cc:504] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.065464 24715 leader_election.cc:288] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405), 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481)
I20260806 10:48:06.073148 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.066058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:35037 (local address 127.16.21.3:36481)
0806 10:48:06.066263 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.066269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.067086 (+ 817us) server_negotiation.cc:407] Connection header received
0806 10:48:06.067338 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.067343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.067431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.067490 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:06.068319 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.069286 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.071089 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.072143 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.072298 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.072398 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.072727 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.072854 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:48:06.073341 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.065691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:36481 (local address 127.16.21.2:35037)
0806 10:48:06.066984 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.067006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.067182 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.067697 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.067701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.067725 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:06.068126 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.068138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.069420 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.069425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.070932 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.070944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.072374 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.072381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.072552 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.072588 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.072686 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.072877 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":22,"thread_start_us":650,"threads_started":1}
I20260806 10:48:06.073968 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a815a3ee804ec784ae59deb672f261" candidate_uuid: "3287a7af1e9943aea787d18b0c885510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ad2656f9b4f4791da216a46876e56" is_pre_election: true
I20260806 10:48:06.074204 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.065869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58755 (local address 127.16.21.1:37405)
0806 10:48:06.066103 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.066112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.066822 (+ 710us) server_negotiation.cc:407] Connection header received
0806 10:48:06.067102 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.067106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.067165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.067264 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:06.067804 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.068889 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.071533 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.072854 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.073009 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.073102 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.073766 (+ 664us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.073922 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:48:06.074194 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.065627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37405 (local address 127.16.21.2:58755)
0806 10:48:06.066706 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.066729 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.066937 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.067298 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.067303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.067329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:06.067621 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.067629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.069031 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.069037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.071351 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.071366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.073323 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.073331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.073531 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.073569 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.073708 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.073844 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":58,"thread_start_us":1033,"threads_started":1}
I20260806 10:48:06.074246 24629 raft_consensus.cc:2454] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3287a7af1e9943aea787d18b0c885510 in term 0.
I20260806 10:48:06.074807 24423 leader_election.cc:302] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3287a7af1e9943aea787d18b0c885510, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:06.075279 24715 raft_consensus.cc:2781] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:06.075167 24348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a815a3ee804ec784ae59deb672f261" candidate_uuid: "3287a7af1e9943aea787d18b0c885510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee5487a8ba645f29546931d691cf583" is_pre_election: true
I20260806 10:48:06.075417 24715 raft_consensus.cc:482] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:06.075515 24715 raft_consensus.cc:3037] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:06.075598 24348 raft_consensus.cc:2454] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3287a7af1e9943aea787d18b0c885510 in term 0.
I20260806 10:48:06.077958 24715 raft_consensus.cc:504] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.078604 24715 leader_election.cc:288] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405), 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481)
I20260806 10:48:06.078976 24348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a815a3ee804ec784ae59deb672f261" candidate_uuid: "3287a7af1e9943aea787d18b0c885510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee5487a8ba645f29546931d691cf583"
I20260806 10:48:06.079038 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a815a3ee804ec784ae59deb672f261" candidate_uuid: "3287a7af1e9943aea787d18b0c885510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ad2656f9b4f4791da216a46876e56"
I20260806 10:48:06.079180 24348 raft_consensus.cc:3037] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:06.079249 24629 raft_consensus.cc:3037] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:06.081588 24629 raft_consensus.cc:2454] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3287a7af1e9943aea787d18b0c885510 in term 1.
I20260806 10:48:06.082027 24423 leader_election.cc:302] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3287a7af1e9943aea787d18b0c885510, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:06.082515 24348 raft_consensus.cc:2454] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3287a7af1e9943aea787d18b0c885510 in term 1.
I20260806 10:48:06.082566 24715 raft_consensus.cc:2781] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:06.088570 24715 raft_consensus.cc:686] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 1 LEADER]: Becoming Leader. State: Replica: 3287a7af1e9943aea787d18b0c885510, State: Running, Role: LEADER
I20260806 10:48:06.088977 24715 consensus_queue.cc:237] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.092536 24190 catalog_manager.cc:5495] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3287a7af1e9943aea787d18b0c885510 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "3287a7af1e9943aea787d18b0c885510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } health_report { overall_health: UNKNOWN } } }
Aug 06 10:48:06 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013283, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.62@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:06.158248 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.138512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46471 (local address 127.0.0.1:52156)
0806 10:48:06.139105 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.139134 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.139305 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.139745 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.139749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.140501 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:06.140692 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.140701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.142173 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.142177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.143082 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.143092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.144036 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.144040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.144173 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.149034 (+ 4861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:06.149081 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:06.151741 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:06.155754 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:06.155765 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:06.155770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:06.156180 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:06.156637 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:06.156640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:06.156643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:06.156810 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:06.157411 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:06.157417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:06.157567 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.157704 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.157928 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260806 10:48:06.158665 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.138573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.16.21.62:46471)
0806 10:48:06.139370 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.139377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.139397 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:48:06.139455 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.139461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.139536 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.139652 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:06.140840 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.142048 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.143225 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.143867 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.143992 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.151966 (+ 7974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:06.151973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:06.151978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:06.152013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:06.155615 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:06.156364 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:06.156370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:06.156373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:06.156530 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:06.156915 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:06.156920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:06.156923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:06.157286 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:06.157438 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.157927 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.158228 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":529,"threads_started":1}
I20260806 10:48:06.164407 24190 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:52156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 10:48:06.165802 24190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:06.173565 24328 tablet_service.cc:1467] Processing CreateTablet for tablet fbc8aa4429554c9483e934d2aba6bba3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfe5a4a18aa94f409cc5f9bed4308ed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:48:06.173066 24469 tablet_service.cc:1467] Processing CreateTablet for tablet fbc8aa4429554c9483e934d2aba6bba3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfe5a4a18aa94f409cc5f9bed4308ed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:48:06.174190 24328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc8aa4429554c9483e934d2aba6bba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:06.174190 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc8aa4429554c9483e934d2aba6bba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:06.176785 24609 tablet_service.cc:1467] Processing CreateTablet for tablet fbc8aa4429554c9483e934d2aba6bba3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfe5a4a18aa94f409cc5f9bed4308ed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:48:06.177265 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc8aa4429554c9483e934d2aba6bba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:06.181515 24709 tablet_bootstrap.cc:492] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: Bootstrap starting.
I20260806 10:48:06.183245 24709 tablet_bootstrap.cc:654] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:06.184429 24708 tablet_bootstrap.cc:492] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: Bootstrap starting.
I20260806 10:48:06.186754 24708 tablet_bootstrap.cc:654] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:06.192147 24707 tablet_bootstrap.cc:492] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: Bootstrap starting.
I20260806 10:48:06.194516 24707 tablet_bootstrap.cc:654] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:06.194835 24708 tablet_bootstrap.cc:492] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: No bootstrap required, opened a new log
I20260806 10:48:06.195016 24708 ts_tablet_manager.cc:1397] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 10:48:06.195403 24709 tablet_bootstrap.cc:492] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: No bootstrap required, opened a new log
I20260806 10:48:06.195694 24709 ts_tablet_manager.cc:1397] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260806 10:48:06.195865 24708 raft_consensus.cc:348] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.196133 24708 raft_consensus.cc:374] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.196225 24708 raft_consensus.cc:729] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.196508 24708 consensus_queue.cc:260] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.196686 24709 raft_consensus.cc:348] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.197013 24709 raft_consensus.cc:374] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.197085 24708 ts_tablet_manager.cc:1428] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:48:06.197098 24709 raft_consensus.cc:729] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3287a7af1e9943aea787d18b0c885510, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.197453 24709 consensus_queue.cc:260] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.198127 24709 ts_tablet_manager.cc:1428] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 10:48:06.197965 24715 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.198266 24715 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.199689 24714 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.200100 24714 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.206012 24707 tablet_bootstrap.cc:492] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: No bootstrap required, opened a new log
I20260806 10:48:06.206252 24707 ts_tablet_manager.cc:1397] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260806 10:48:06.206904 24707 raft_consensus.cc:348] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.207089 24707 raft_consensus.cc:374] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.207197 24707 raft_consensus.cc:729] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.207437 24707 consensus_queue.cc:260] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.207836 24718 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.208060 24707 ts_tablet_manager.cc:1428] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 10:48:06.208210 24718 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.341369 24718 raft_consensus.cc:482] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:06.341620 24718 raft_consensus.cc:504] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.342386 24718 leader_election.cc:288] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079), 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481)
I20260806 10:48:06.342928 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc8aa4429554c9483e934d2aba6bba3" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510" is_pre_election: true
I20260806 10:48:06.343060 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc8aa4429554c9483e934d2aba6bba3" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ad2656f9b4f4791da216a46876e56" is_pre_election: true
I20260806 10:48:06.343195 24489 raft_consensus.cc:2454] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 0.
I20260806 10:48:06.343380 24629 raft_consensus.cc:2454] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 0.
I20260806 10:48:06.343721 24280 leader_election.cc:302] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee5487a8ba645f29546931d691cf583, 3287a7af1e9943aea787d18b0c885510; no voters:
I20260806 10:48:06.344048 24718 raft_consensus.cc:2781] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:06.344158 24718 raft_consensus.cc:482] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:06.344328 24718 raft_consensus.cc:3037] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:06.346720 24718 raft_consensus.cc:504] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.347301 24718 leader_election.cc:288] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 election: Requested vote from peers 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079), 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481)
I20260806 10:48:06.347800 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc8aa4429554c9483e934d2aba6bba3" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510"
I20260806 10:48:06.347801 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc8aa4429554c9483e934d2aba6bba3" candidate_uuid: "1ee5487a8ba645f29546931d691cf583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ad2656f9b4f4791da216a46876e56"
I20260806 10:48:06.348013 24489 raft_consensus.cc:3037] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:06.348033 24629 raft_consensus.cc:3037] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:06.351058 24629 raft_consensus.cc:2454] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 1.
I20260806 10:48:06.351075 24489 raft_consensus.cc:2454] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee5487a8ba645f29546931d691cf583 in term 1.
I20260806 10:48:06.351456 24280 leader_election.cc:302] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee5487a8ba645f29546931d691cf583, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:06.351809 24718 raft_consensus.cc:2781] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:06.351998 24718 raft_consensus.cc:686] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Running, Role: LEADER
I20260806 10:48:06.352384 24718 consensus_queue.cc:237] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } }
I20260806 10:48:06.354436 24727 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: TxnStatusTablet state changed. Reason: New leader 1ee5487a8ba645f29546931d691cf583. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.354918 24727 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:48:06.355814 24753 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 10:48:06.356056 24753 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 10:48:06.355980 24190 catalog_manager.cc:5495] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ee5487a8ba645f29546931d691cf583 (127.16.21.1). New cstate: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } health_report { overall_health: HEALTHY } } }
I20260806 10:48:06.386942 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.381071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37405 (local address 127.0.0.1:46846)
0806 10:48:06.381314 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.381335 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.381478 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.381946 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.381950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.381960 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:48:06.382103 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.382110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.383698 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.383702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.384786 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.384798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.385871 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.385876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.386029 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.386081 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.386534 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.386549 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.386606 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.386712 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:48:06.387042 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.381081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.16.21.1:37405)
0806 10:48:06.381289 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.381295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.381415 (+ 120us) server_negotiation.cc:407] Connection header received
0806 10:48:06.381660 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.381666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.381785 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.381903 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:06.382246 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.383554 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.384933 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.385726 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.385874 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.386282 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.386424 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.386516 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.386669 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.386750 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:48:06.390393 24489 raft_consensus.cc:1261] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Refusing update from remote peer 1ee5487a8ba645f29546931d691cf583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:06.390554 24629 raft_consensus.cc:1261] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Refusing update from remote peer 1ee5487a8ba645f29546931d691cf583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:06.391057 24718 consensus_queue.cc:1035] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:06.391350 24727 consensus_queue.cc:1035] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:06.394965 24714 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: TxnStatusTablet state changed. Reason: New leader 1ee5487a8ba645f29546931d691cf583. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.395264 24714 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.397240 24713 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.397574 24713 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:48:06.399504 24714 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.399726 24714 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 500ad2656f9b4f4791da216a46876e56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.406107 24715 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: TxnStatusTablet state changed. Reason: New leader 1ee5487a8ba645f29546931d691cf583. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.406453 24715 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.407949 24727 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.408211 24727 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 1ee5487a8ba645f29546931d691cf583: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:48:06.410960 24715 tablet_replica.cc:439] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ee5487a8ba645f29546931d691cf583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } } peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } }
I20260806 10:48:06.411248 24715 tablet_replica.cc:442] T fbc8aa4429554c9483e934d2aba6bba3 P 3287a7af1e9943aea787d18b0c885510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:48:06.432413 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.417661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.16.21.2:46079)
0806 10:48:06.418273 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.418281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.424919 (+ 6638us) server_negotiation.cc:407] Connection header received
0806 10:48:06.425118 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.425124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.425220 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.425384 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:06.425985 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.427169 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.429373 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.430026 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.430142 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.430825 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.431548 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.431721 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.431860 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.431950 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":291,"threads_started":1}
I20260806 10:48:06.455379 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.444286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:47421 (local address 127.16.21.1:37405)
0806 10:48:06.444475 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.444481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.448938 (+ 4457us) server_negotiation.cc:407] Connection header received
0806 10:48:06.449134 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.449140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.449226 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.449326 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:06.450047 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.451216 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.453569 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.454041 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.454139 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.454704 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.454835 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.454901 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.455076 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.455146 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:48:06.455660 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.444107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37405 (local address 127.16.21.2:47421)
0806 10:48:06.448814 (+ 4707us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.448836 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.448993 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.449427 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.449433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.449447 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:48:06.449856 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.449868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.451405 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.451411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.453391 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.453409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.454244 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.454254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.454456 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.454534 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.454939 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.454952 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.455051 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.455223 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4524,"thread_start_us":4369,"threads_started":1}
I20260806 10:48:06.466789 24348 raft_consensus.cc:1261] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Refusing update from remote peer 3287a7af1e9943aea787d18b0c885510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:06.467458 24629 raft_consensus.cc:1261] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Refusing update from remote peer 3287a7af1e9943aea787d18b0c885510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:06.467620 24739 consensus_queue.cc:1035] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:06.468113 24715 consensus_queue.cc:1035] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:06.524796 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.508933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:58383 (local address 127.16.21.2:46079)
0806 10:48:06.509106 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.509112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.518004 (+ 8892us) server_negotiation.cc:407] Connection header received
0806 10:48:06.518315 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.518321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.518407 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.518469 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:06.519228 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.520385 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.522480 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.523100 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.523200 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.523904 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.524039 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.524103 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.524388 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.524476 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:48:06.525856 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.510758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46079 (local address 127.16.21.1:58383)
0806 10:48:06.517891 (+ 7133us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.517914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.518134 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.518672 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.518677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.518690 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:48:06.519043 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.519054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.520559 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.520565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.522302 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.522319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.523401 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.523407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.523612 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.523696 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:48:06.524184 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:48:06.524201 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.525254 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.525414 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6888,"thread_start_us":8172,"threads_started":1}
I20260806 10:48:06.562778 24190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:06.563051 24190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:06.566731 24190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Sending DeleteTablet for 3 replicas of tablet 18a815a3ee804ec784ae59deb672f261
I20260806 10:48:06.567765 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a815a3ee804ec784ae59deb672f261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:06 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260806 10:48:06.567777 24609 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a815a3ee804ec784ae59deb672f261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:06 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41866
I20260806 10:48:06.568591 24328 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a815a3ee804ec784ae59deb672f261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:06 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:46832
I20260806 10:48:06.583339 24771 tablet_replica.cc:330] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: stopping tablet replica
I20260806 10:48:06.583581 24771 raft_consensus.cc:2229] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:06.583824 24771 raft_consensus.cc:2258] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:06.585072 24771 ts_tablet_manager.cc:1905] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:06.592800 24771 ts_tablet_manager.cc:1918] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:48:06.592947 24771 log.cc:1196] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/18a815a3ee804ec784ae59deb672f261
I20260806 10:48:06.593400 24771 ts_tablet_manager.cc:1939] T 18a815a3ee804ec784ae59deb672f261 P 500ad2656f9b4f4791da216a46876e56: Deleting consensus metadata
I20260806 10:48:06.594612 24177 catalog_manager.cc:4841] TS 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481): tablet 18a815a3ee804ec784ae59deb672f261 (table test-table [id=03a7926e7c7b40beaddfd8f70d1a8ce0]) successfully deleted
I20260806 10:48:06.604791 24772 tablet_replica.cc:330] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: stopping tablet replica
I20260806 10:48:06.605033 24772 raft_consensus.cc:2229] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:06.605355 24772 raft_consensus.cc:2258] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:06.606045 24770 tablet_replica.cc:330] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: stopping tablet replica
I20260806 10:48:06.606290 24770 raft_consensus.cc:2229] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:48:06.606854 24772 ts_tablet_manager.cc:1905] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:06.606914 24770 raft_consensus.cc:2258] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:06.608016 24770 ts_tablet_manager.cc:1905] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:06.615882 24770 ts_tablet_manager.cc:1918] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:48:06.616030 24770 log.cc:1196] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/18a815a3ee804ec784ae59deb672f261
I20260806 10:48:06.616577 24770 ts_tablet_manager.cc:1939] T 18a815a3ee804ec784ae59deb672f261 P 3287a7af1e9943aea787d18b0c885510: Deleting consensus metadata
I20260806 10:48:06.617799 24177 catalog_manager.cc:4841] TS 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079): tablet 18a815a3ee804ec784ae59deb672f261 (table test-table [id=03a7926e7c7b40beaddfd8f70d1a8ce0]) successfully deleted
I20260806 10:48:06.621279 24772 ts_tablet_manager.cc:1918] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:48:06.621500 24772 log.cc:1196] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/18a815a3ee804ec784ae59deb672f261
I20260806 10:48:06.622114 24772 ts_tablet_manager.cc:1939] T 18a815a3ee804ec784ae59deb672f261 P 1ee5487a8ba645f29546931d691cf583: Deleting consensus metadata
I20260806 10:48:06.623476 24174 catalog_manager.cc:4841] TS 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405): tablet 18a815a3ee804ec784ae59deb672f261 (table test-table [id=03a7926e7c7b40beaddfd8f70d1a8ce0]) successfully deleted
Aug 06 10:48:06 dist-test-slave-0b58 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:48:06 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:06 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:48:06.697530 24781 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:06.671625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.16.21.62:46471)
0806 10:48:06.683940 (+ 12315us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.683947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.684624 (+ 677us) server_negotiation.cc:407] Connection header received
0806 10:48:06.684829 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.684836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.684939 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.685068 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:06.686905 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.688125 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.690017 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.690715 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.690854 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.697174 (+ 6320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52160: BlockingRecv error: recv got EOF from 127.0.0.1:52160 (error 108)
Metrics: {"server-negotiator.queue_time_us":12155,"thread_start_us":12316,"threads_started":1}
Aug 06 10:48:06 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM
I20260806 10:48:06.719509 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.701393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.16.21.62:46471)
0806 10:48:06.701570 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.701576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.702286 (+ 710us) server_negotiation.cc:407] Connection header received
0806 10:48:06.702487 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.702494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.702573 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.702639 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:06.704348 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.705443 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.707294 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.707972 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.708101 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.713073 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:06.713080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:06.713085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:06.713123 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:06.716577 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:06.717365 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:06.717370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:06.717373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:06.717538 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:06.717991 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:06.717996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:06.717999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:06.718356 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:06.718609 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.718949 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.719169 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:48:06.723886 24190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:48:06.724689 24190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:06.732966 24469 tablet_service.cc:1467] Processing CreateTablet for tablet c4f02baf64cc4519be86645733d1bf24 (DEFAULT_TABLE table=test-table [id=051bef73eff44795b3dc6bb46fdae8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:06.733479 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f02baf64cc4519be86645733d1bf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:06.736948 24609 tablet_service.cc:1467] Processing CreateTablet for tablet c4f02baf64cc4519be86645733d1bf24 (DEFAULT_TABLE table=test-table [id=051bef73eff44795b3dc6bb46fdae8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:06.737412 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f02baf64cc4519be86645733d1bf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:06.738462 24328 tablet_service.cc:1467] Processing CreateTablet for tablet c4f02baf64cc4519be86645733d1bf24 (DEFAULT_TABLE table=test-table [id=051bef73eff44795b3dc6bb46fdae8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:06.738916 24328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f02baf64cc4519be86645733d1bf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:06.753099 24787 tablet_bootstrap.cc:492] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Bootstrap starting.
I20260806 10:48:06.754885 24787 tablet_bootstrap.cc:654] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:06.767402 24789 tablet_bootstrap.cc:492] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Bootstrap starting.
I20260806 10:48:06.770891 24788 tablet_bootstrap.cc:492] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Bootstrap starting.
I20260806 10:48:06.773488 24789 tablet_bootstrap.cc:654] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:06.776078 24788 tablet_bootstrap.cc:654] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:06.779868 24789 tablet_bootstrap.cc:492] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: No bootstrap required, opened a new log
I20260806 10:48:06.780484 24789 ts_tablet_manager.cc:1397] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20260806 10:48:06.781670 24787 tablet_bootstrap.cc:492] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: No bootstrap required, opened a new log
I20260806 10:48:06.782308 24788 tablet_bootstrap.cc:492] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: No bootstrap required, opened a new log
I20260806 10:48:06.782483 24788 ts_tablet_manager.cc:1397] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260806 10:48:06.782625 24787 ts_tablet_manager.cc:1397] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.020s
I20260806 10:48:06.783344 24788 raft_consensus.cc:348] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.783674 24788 raft_consensus.cc:374] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.783790 24788 raft_consensus.cc:729] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3287a7af1e9943aea787d18b0c885510, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.784101 24788 consensus_queue.cc:260] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.784173 24789 raft_consensus.cc:348] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.784629 24789 raft_consensus.cc:374] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.783489 24787 raft_consensus.cc:348] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.785622 24789 raft_consensus.cc:729] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.785806 24787 raft_consensus.cc:374] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:06.785952 24787 raft_consensus.cc:729] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Initialized, Role: FOLLOWER
I20260806 10:48:06.786273 24789 consensus_queue.cc:260] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.786769 24787 consensus_queue.cc:260] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.788726 24787 ts_tablet_manager.cc:1428] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20260806 10:48:06.788225 24789 ts_tablet_manager.cc:1428] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260806 10:48:06.790342 24788 ts_tablet_manager.cc:1428] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260806 10:48:06.974401 24714 raft_consensus.cc:482] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:06.974680 24714 raft_consensus.cc:504] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:06.976541 24714 leader_election.cc:288] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405), 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:06.999325 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.977784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:51111 (local address 127.16.21.2:46079)
0806 10:48:06.977951 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.977957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.992509 (+ 14552us) server_negotiation.cc:407] Connection header received
0806 10:48:06.992796 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.992803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.992887 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.993009 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:06.993442 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.994505 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.996766 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.998255 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.998465 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.998576 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.998897 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.999039 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:48:06.999737 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.977435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46079 (local address 127.16.21.3:51111)
0806 10:48:06.992410 (+ 14975us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.992427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.992618 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.993030 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.993033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.993054 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:06.993264 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.993271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.994651 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.994656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.996589 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.996605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.998433 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.998440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.998630 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.998670 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.998885 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.999029 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14858,"spinlock_wait_cycles":1792,"thread_start_us":14875,"threads_started":1}
I20260806 10:48:07.000293 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.976805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37405 (local address 127.16.21.3:60165)
0806 10:48:06.992184 (+ 15379us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:06.992218 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:06.992618 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:06.993479 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:06.993485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:06.993518 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:06.993894 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.993907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.995806 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.995812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.997416 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:06.997428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.998813 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.998821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:06.998998 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.999032 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:48:06.999138 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 10:48:06.999275 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15185,"spinlock_wait_cycles":15616,"thread_start_us":15283,"threads_started":1}
I20260806 10:48:07.000280 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:06.976949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:60165 (local address 127.16.21.1:37405)
0806 10:48:06.987540 (+ 10591us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:06.987546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:06.992413 (+ 4867us) server_negotiation.cc:407] Connection header received
0806 10:48:06.993239 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:06.993248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:06.993348 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:06.993456 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:06.994073 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.995329 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.997572 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:06.998526 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:06.998658 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:06.998761 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:06.999802 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0806 10:48:06.999960 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10452,"thread_start_us":10341,"threads_started":1}
I20260806 10:48:07.000556 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f02baf64cc4519be86645733d1bf24" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510" is_pre_election: true
I20260806 10:48:07.000854 24489 raft_consensus.cc:2454] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 0.
I20260806 10:48:07.001324 24347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f02baf64cc4519be86645733d1bf24" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee5487a8ba645f29546931d691cf583" is_pre_election: true
I20260806 10:48:07.001430 24564 leader_election.cc:302] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3287a7af1e9943aea787d18b0c885510, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:07.001616 24347 raft_consensus.cc:2454] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 0.
I20260806 10:48:07.001876 24714 raft_consensus.cc:2781] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:07.002018 24714 raft_consensus.cc:482] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:07.002156 24714 raft_consensus.cc:3037] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:07.004477 24714 raft_consensus.cc:504] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.005117 24714 leader_election.cc:288] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405), 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:07.005656 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f02baf64cc4519be86645733d1bf24" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510"
I20260806 10:48:07.005692 24347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f02baf64cc4519be86645733d1bf24" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee5487a8ba645f29546931d691cf583"
I20260806 10:48:07.005882 24489 raft_consensus.cc:3037] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:07.005919 24347 raft_consensus.cc:3037] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:07.008348 24489 raft_consensus.cc:2454] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 1.
I20260806 10:48:07.008826 24564 leader_election.cc:302] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3287a7af1e9943aea787d18b0c885510, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:07.009147 24714 raft_consensus.cc:2781] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:07.009256 24347 raft_consensus.cc:2454] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 1.
I20260806 10:48:07.009682 24714 raft_consensus.cc:686] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 1 LEADER]: Becoming Leader. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Running, Role: LEADER
I20260806 10:48:07.010102 24714 consensus_queue.cc:237] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.013981 24189 catalog_manager.cc:5495] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 500ad2656f9b4f4791da216a46876e56 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "500ad2656f9b4f4791da216a46876e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } health_report { overall_health: HEALTHY } } }
I20260806 10:48:07.055815 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.048889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.16.21.3:36481)
0806 10:48:07.049519 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:07.049525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:07.049552 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:48:07.049663 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:07.049669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:07.049810 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:07.050014 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:07.050308 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.051511 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.053171 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.053808 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.053929 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.054447 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:07.055164 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:07.055313 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:07.055372 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 10:48:07.055451 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":387,"threads_started":1}
I20260806 10:48:07.060020 24489 raft_consensus.cc:1261] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Refusing update from remote peer 500ad2656f9b4f4791da216a46876e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:07.060683 24347 raft_consensus.cc:1261] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Refusing update from remote peer 500ad2656f9b4f4791da216a46876e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:07.060791 24796 consensus_queue.cc:1035] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:07.061332 24714 consensus_queue.cc:1035] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:07.095381 24189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:07.095621 24189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:07.098656 24189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Sending DeleteTablet for 3 replicas of tablet c4f02baf64cc4519be86645733d1bf24
I20260806 10:48:07.099407 24328 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f02baf64cc4519be86645733d1bf24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:07 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:46832
I20260806 10:48:07.099488 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f02baf64cc4519be86645733d1bf24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:07 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260806 10:48:07.099682 24772 tablet_replica.cc:330] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: stopping tablet replica
I20260806 10:48:07.099880 24772 raft_consensus.cc:2229] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:07.100117 24772 raft_consensus.cc:2258] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:07.100417 24770 tablet_replica.cc:330] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: stopping tablet replica
I20260806 10:48:07.101104 24609 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f02baf64cc4519be86645733d1bf24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:07 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41866
I20260806 10:48:07.101202 24772 ts_tablet_manager.cc:1905] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:07.101507 24770 raft_consensus.cc:2229] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:07.101734 24770 raft_consensus.cc:2258] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:07.102989 24770 ts_tablet_manager.cc:1905] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:07.104130 24189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:48:07.105470 24189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:07.108322 24772 ts_tablet_manager.cc:1918] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:07.108492 24772 log.cc:1196] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/c4f02baf64cc4519be86645733d1bf24
I20260806 10:48:07.108960 24772 ts_tablet_manager.cc:1939] T c4f02baf64cc4519be86645733d1bf24 P 1ee5487a8ba645f29546931d691cf583: Deleting consensus metadata
I20260806 10:48:07.110167 24174 catalog_manager.cc:4841] TS 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405): tablet c4f02baf64cc4519be86645733d1bf24 (table test-table [id=051bef73eff44795b3dc6bb46fdae8e9]) successfully deleted
I20260806 10:48:07.112703 24770 ts_tablet_manager.cc:1918] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:07.112869 24770 log.cc:1196] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/c4f02baf64cc4519be86645733d1bf24
I20260806 10:48:07.113289 24770 ts_tablet_manager.cc:1939] T c4f02baf64cc4519be86645733d1bf24 P 3287a7af1e9943aea787d18b0c885510: Deleting consensus metadata
I20260806 10:48:07.114719 24177 catalog_manager.cc:4841] TS 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079): tablet c4f02baf64cc4519be86645733d1bf24 (table test-table [id=051bef73eff44795b3dc6bb46fdae8e9]) successfully deleted
I20260806 10:48:07.115396 24469 tablet_service.cc:1467] Processing CreateTablet for tablet 5c611583e1d14678b1662f0da74883e9 (DEFAULT_TABLE table=test-table [id=ff8309e2597f409ba5f479bf56c8f97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:07.115530 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 5c611583e1d14678b1662f0da74883e9 (DEFAULT_TABLE table=test-table [id=ff8309e2597f409ba5f479bf56c8f97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:07.115796 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c611583e1d14678b1662f0da74883e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:07.116019 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c611583e1d14678b1662f0da74883e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:07.120707 24810 tablet_replica.cc:330] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: stopping tablet replica
I20260806 10:48:07.121001 24810 raft_consensus.cc:2229] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:48:07.120998 24328 tablet_service.cc:1467] Processing CreateTablet for tablet 5c611583e1d14678b1662f0da74883e9 (DEFAULT_TABLE table=test-table [id=ff8309e2597f409ba5f479bf56c8f97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:07.121433 24328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c611583e1d14678b1662f0da74883e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:07.123139 24810 raft_consensus.cc:2258] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:07.123816 24787 tablet_bootstrap.cc:492] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Bootstrap starting.
I20260806 10:48:07.125280 24787 tablet_bootstrap.cc:654] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:07.127446 24787 tablet_bootstrap.cc:492] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: No bootstrap required, opened a new log
I20260806 10:48:07.127655 24787 ts_tablet_manager.cc:1397] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 10:48:07.128468 24789 tablet_bootstrap.cc:492] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Bootstrap starting.
I20260806 10:48:07.128417 24787 raft_consensus.cc:348] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.128674 24787 raft_consensus.cc:374] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:07.128756 24787 raft_consensus.cc:729] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Initialized, Role: FOLLOWER
I20260806 10:48:07.129000 24787 consensus_queue.cc:260] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.129685 24787 ts_tablet_manager.cc:1428] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:48:07.130105 24789 tablet_bootstrap.cc:654] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:07.131242 24788 tablet_bootstrap.cc:492] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Bootstrap starting.
I20260806 10:48:07.132933 24788 tablet_bootstrap.cc:654] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:07.133155 24789 tablet_bootstrap.cc:492] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: No bootstrap required, opened a new log
I20260806 10:48:07.133327 24789 ts_tablet_manager.cc:1397] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260806 10:48:07.134040 24789 raft_consensus.cc:348] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.134301 24789 raft_consensus.cc:374] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:07.134383 24789 raft_consensus.cc:729] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee5487a8ba645f29546931d691cf583, State: Initialized, Role: FOLLOWER
I20260806 10:48:07.134641 24789 consensus_queue.cc:260] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.135752 24788 tablet_bootstrap.cc:492] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: No bootstrap required, opened a new log
I20260806 10:48:07.135926 24788 ts_tablet_manager.cc:1397] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 10:48:07.136632 24788 raft_consensus.cc:348] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.136895 24788 raft_consensus.cc:374] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:07.136973 24788 raft_consensus.cc:729] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3287a7af1e9943aea787d18b0c885510, State: Initialized, Role: FOLLOWER
I20260806 10:48:07.137230 24788 consensus_queue.cc:260] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.137351 24789 ts_tablet_manager.cc:1428] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260806 10:48:07.138413 24788 ts_tablet_manager.cc:1428] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:48:07.138141 24810 ts_tablet_manager.cc:1905] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:07.144963 24810 ts_tablet_manager.cc:1918] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:07.145110 24810 log.cc:1196] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/c4f02baf64cc4519be86645733d1bf24
I20260806 10:48:07.145581 24810 ts_tablet_manager.cc:1939] T c4f02baf64cc4519be86645733d1bf24 P 500ad2656f9b4f4791da216a46876e56: Deleting consensus metadata
I20260806 10:48:07.146528 24177 catalog_manager.cc:4841] TS 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481): tablet c4f02baf64cc4519be86645733d1bf24 (table test-table [id=051bef73eff44795b3dc6bb46fdae8e9]) successfully deleted
I20260806 10:48:07.179342 24807 raft_consensus.cc:482] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:07.179586 24807 raft_consensus.cc:504] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.180492 24807 leader_election.cc:288] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405), 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:07.181015 24347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c611583e1d14678b1662f0da74883e9" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee5487a8ba645f29546931d691cf583" is_pre_election: true
I20260806 10:48:07.181103 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c611583e1d14678b1662f0da74883e9" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510" is_pre_election: true
I20260806 10:48:07.181267 24347 raft_consensus.cc:2454] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 0.
I20260806 10:48:07.181349 24489 raft_consensus.cc:2454] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 0.
I20260806 10:48:07.181810 24561 leader_election.cc:302] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee5487a8ba645f29546931d691cf583, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:07.182130 24807 raft_consensus.cc:2781] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:07.182271 24807 raft_consensus.cc:482] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:07.182387 24807 raft_consensus.cc:3037] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:07.185034 24807 raft_consensus.cc:504] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.185674 24807 leader_election.cc:288] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405), 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079)
I20260806 10:48:07.186071 24347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c611583e1d14678b1662f0da74883e9" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee5487a8ba645f29546931d691cf583"
I20260806 10:48:07.186245 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c611583e1d14678b1662f0da74883e9" candidate_uuid: "500ad2656f9b4f4791da216a46876e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3287a7af1e9943aea787d18b0c885510"
I20260806 10:48:07.186357 24347 raft_consensus.cc:3037] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:07.186466 24489 raft_consensus.cc:3037] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:07.188654 24347 raft_consensus.cc:2454] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 1.
I20260806 10:48:07.189117 24561 leader_election.cc:302] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee5487a8ba645f29546931d691cf583, 500ad2656f9b4f4791da216a46876e56; no voters:
I20260806 10:48:07.189473 24807 raft_consensus.cc:2781] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:07.189507 24489 raft_consensus.cc:2454] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500ad2656f9b4f4791da216a46876e56 in term 1.
I20260806 10:48:07.190119 24807 raft_consensus.cc:686] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 1 LEADER]: Becoming Leader. State: Replica: 500ad2656f9b4f4791da216a46876e56, State: Running, Role: LEADER
I20260806 10:48:07.190487 24807 consensus_queue.cc:237] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } }
I20260806 10:48:07.193348 24188 catalog_manager.cc:5495] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 500ad2656f9b4f4791da216a46876e56 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "500ad2656f9b4f4791da216a46876e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500ad2656f9b4f4791da216a46876e56" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36481 } health_report { overall_health: HEALTHY } } }
I20260806 10:48:07.228611 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.222201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:45113 (local address 127.16.21.1:37405)
0806 10:48:07.222414 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:07.222419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:07.223087 (+ 668us) server_negotiation.cc:407] Connection header received
0806 10:48:07.223296 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:07.223302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:07.223391 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:07.223487 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:07.223997 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.225147 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.226628 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.227329 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.227460 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.227870 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:07.228001 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:07.228098 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:07.228233 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 10:48:07.228343 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 10:48:07.228631 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.222107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37405 (local address 127.16.21.3:45113)
0806 10:48:07.222984 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:07.223000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:07.223123 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:07.223509 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:07.223512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:07.223521 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:48:07.223836 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:07.223843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.225268 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.225271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:07.226484 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:07.226495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.227509 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.227514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:07.227646 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.227697 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:48:07.228106 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:48:07.228114 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:48:07.228174 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 10:48:07.228345 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":495,"threads_started":1}
I20260806 10:48:07.237185 24347 raft_consensus.cc:1261] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Refusing update from remote peer 500ad2656f9b4f4791da216a46876e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:07.237321 24489 raft_consensus.cc:1261] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Refusing update from remote peer 500ad2656f9b4f4791da216a46876e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:07.237859 24714 consensus_queue.cc:1035] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee5487a8ba645f29546931d691cf583" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:07.238121 24807 consensus_queue.cc:1035] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3287a7af1e9943aea787d18b0c885510" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:07.281486 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.274832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:51387 (local address 127.16.21.3:36481)
0806 10:48:07.275139 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:07.275146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:07.275168 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:48:07.275325 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:07.275331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:07.275419 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:07.275505 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:07.276115 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.277286 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.279358 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.279930 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.280077 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.280732 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:07.280897 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:07.280997 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:07.281143 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 10:48:07.281205 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260806 10:48:07.281886 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.274463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:36481 (local address 127.16.21.1:51387)
0806 10:48:07.275021 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:07.275043 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:07.275169 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:07.275709 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:07.275714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:07.275727 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:48:07.275942 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:07.275957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.277463 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.277469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:07.279156 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:07.279173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.280118 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.280124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:07.280354 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.280406 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:48:07.281017 (+ 611us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:48:07.281032 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 10:48:07.281130 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:48:07.281299 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":418,"threads_started":1}
I20260806 10:48:07.315337 24188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:07.315613 24188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:07.318557 24188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d73d96537843af9b314195c83f4df0: Sending DeleteTablet for 3 replicas of tablet 5c611583e1d14678b1662f0da74883e9
I20260806 10:48:07.319288 24328 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c611583e1d14678b1662f0da74883e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:07 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:46832
I20260806 10:48:07.319422 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c611583e1d14678b1662f0da74883e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:07 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260806 10:48:07.319588 24772 tablet_replica.cc:330] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: stopping tablet replica
I20260806 10:48:07.319737 24770 tablet_replica.cc:330] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: stopping tablet replica
I20260806 10:48:07.319777 24772 raft_consensus.cc:2229] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:07.319950 24770 raft_consensus.cc:2229] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:07.319989 24772 raft_consensus.cc:2258] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:07.320154 24770 raft_consensus.cc:2258] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:07.320534 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c611583e1d14678b1662f0da74883e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:07 UTC) from {username='oryx', principal='oryx/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41866
I20260806 10:48:07.320840 24810 tablet_replica.cc:330] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: stopping tablet replica
I20260806 10:48:07.321048 24810 raft_consensus.cc:2229] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:48:07.321396 24770 ts_tablet_manager.cc:1905] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:07.321398 24810 raft_consensus.cc:2258] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:07.321615 24772 ts_tablet_manager.cc:1905] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:07.323504 24810 ts_tablet_manager.cc:1905] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:07.328238 24770 ts_tablet_manager.cc:1918] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:48:07.328306 24772 ts_tablet_manager.cc:1918] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:48:07.328387 24770 log.cc:1196] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/5c611583e1d14678b1662f0da74883e9
I20260806 10:48:07.328440 24772 log.cc:1196] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/5c611583e1d14678b1662f0da74883e9
I20260806 10:48:07.328640 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 24261
I20260806 10:48:07.328794 24770 ts_tablet_manager.cc:1939] T 5c611583e1d14678b1662f0da74883e9 P 3287a7af1e9943aea787d18b0c885510: Deleting consensus metadata
I20260806 10:48:07.328918 24772 ts_tablet_manager.cc:1939] T 5c611583e1d14678b1662f0da74883e9 P 1ee5487a8ba645f29546931d691cf583: Deleting consensus metadata
I20260806 10:48:07.329814 24177 catalog_manager.cc:4841] TS 3287a7af1e9943aea787d18b0c885510 (127.16.21.2:46079): tablet 5c611583e1d14678b1662f0da74883e9 (table test-table [id=ff8309e2597f409ba5f479bf56c8f97d]) successfully deleted
I20260806 10:48:07.329993 24174 catalog_manager.cc:4841] TS 1ee5487a8ba645f29546931d691cf583 (127.16.21.1:37405): tablet 5c611583e1d14678b1662f0da74883e9 (table test-table [id=ff8309e2597f409ba5f479bf56c8f97d]) successfully deleted
I20260806 10:48:07.333196 24810 ts_tablet_manager.cc:1918] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:48:07.333374 24810 log.cc:1196] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/5c611583e1d14678b1662f0da74883e9
I20260806 10:48:07.333798 24810 ts_tablet_manager.cc:1939] T 5c611583e1d14678b1662f0da74883e9 P 500ad2656f9b4f4791da216a46876e56: Deleting consensus metadata
I20260806 10:48:07.334789 24177 catalog_manager.cc:4841] TS 500ad2656f9b4f4791da216a46876e56 (127.16.21.3:36481): tablet 5c611583e1d14678b1662f0da74883e9 (table test-table [id=ff8309e2597f409ba5f479bf56c8f97d]) successfully deleted
Aug 06 10:48:07 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.1@KRBTEST.COM
I20260806 10:48:07.346834 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.330360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.16.21.1:37405)
0806 10:48:07.330544 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:07.330550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:07.330569 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:48:07.330713 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:07.330719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:07.330792 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:07.330856 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:07.332672 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.333876 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.335681 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.336062 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.336155 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.341042 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:07.341057 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:07.341071 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:07.341107 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:07.344009 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:07.344722 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:07.344735 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:07.344737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:07.344896 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:07.345301 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:07.345305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:07.345307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:07.345625 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:07.345972 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:07.346266 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 10:48:07.346501 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:48:07.347451 24393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:07.541877 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 24261
I20260806 10:48:07.573237 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 24403
Aug 06 10:48:07 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.2@KRBTEST.COM
I20260806 10:48:07.589236 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.574113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.16.21.2:46079)
0806 10:48:07.574630 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:07.574636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:07.574656 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:48:07.574716 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:07.574720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:07.574792 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:07.574915 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:07.576164 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.577057 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.578417 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.578840 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.578974 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.583334 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:07.583349 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:07.583363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:07.583395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:07.586344 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:07.587211 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:07.587218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:07.587220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:07.587354 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:07.587800 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:07.587804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:07.587806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:07.588145 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:07.588545 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:07.588740 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 10:48:07.588922 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260806 10:48:07.589911 24536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:07.716940 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 24403
I20260806 10:48:07.743928 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 24545
Aug 06 10:48:07 dist-test-slave-0b58 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.21.3@KRBTEST.COM
I20260806 10:48:07.759374 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:07.744779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.16.21.3:36481)
0806 10:48:07.744967 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:07.744971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:07.744989 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:48:07.745143 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:07.745147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:07.745208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:07.745308 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:07.746518 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.747180 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.748512 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:07.748915 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:07.749052 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:07.753643 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:07.753659 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:07.753675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:07.753712 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:07.756643 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:07.757328 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:07.757335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:07.757337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:07.757473 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:07.757849 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:07.757856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:07.757859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:07.758200 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:07.758533 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:07.758824 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 10:48:07.759045 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:48:07.760108 24677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:07.887557 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 24545
I20260806 10:48:07.913646 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 24158
I20260806 10:48:07.914785 24230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:08.035791 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 24158
2026-08-06T10:48:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6037 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:48:08 dist-test-slave-0b58 krb5kdc[24834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:48:08 dist-test-slave-0b58 krb5kdc[24834](info): set up 2 sockets
Aug 06 10:48:08 dist-test-slave-0b58 krb5kdc[24834](info): commencing operation
krb5kdc: starting...
W20260806 10:48:09.169312 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:48:09 dist-test-slave-0b58 krb5kdc[24834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:48:09.448587 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45563
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:09.780068 24847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:09.780393 24847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:09.780493 24847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:09.787546 24847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:09.787636 24847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:09.787683 24847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:09.787732 24847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:09.787781 24847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:09.797279 24847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:09.798388 24847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:09.799551 24847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:09.811412 24852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:09.811372 24855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:09.811466 24853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:09.812239 24847 server_base.cc:1034] running on GCE node
I20260806 10:48:09.813016 24847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:09.814302 24847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:09.815528 24847 hybrid_clock.cc:648] HybridClock initialized: now 1786013289815483 us; error 56 us; skew 500 ppm
I20260806 10:48:09.815857 24847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T10:48:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1850 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:48:10 dist-test-slave-0b58 krb5kdc[24861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:48:10 dist-test-slave-0b58 krb5kdc[24861](info): set up 2 sockets
Aug 06 10:48:10 dist-test-slave-0b58 krb5kdc[24861](info): commencing operation
krb5kdc: starting...
W20260806 10:48:11.013010 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.009s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:48:11 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:48:11.259197 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35175
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:11.584683 24874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:11.585028 24874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:11.585207 24874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:11.592417 24874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:11.592571 24874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:11.592679 24874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:11.592792 24874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:11.592875 24874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:11.602550 24874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:11.603881 24874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:11.605238 24874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:11.616830 24879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:11.616881 24882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:11.616909 24880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:11.618376 24874 server_base.cc:1034] running on GCE node
I20260806 10:48:11.619079 24874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:11.620437 24874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:11.621668 24874 hybrid_clock.cc:648] HybridClock initialized: now 1786013291621629 us; error 31 us; skew 500 ppm
I20260806 10:48:11.622020 24874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:11 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013291, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:11.628507 24874 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:48:11.630033 24874 webserver.cc:460] Webserver started at http://127.16.21.62:45149/ using document root <none> and password file <none>
I20260806 10:48:11.630717 24874 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:11.630852 24874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:11.631173 24874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:11.634177 24874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "50118dab0f804596a404eb955da7bac0"
format_stamp: "Formatted at 2026-08-06 10:48:11 on dist-test-slave-0b58"
server_key: "78c115b26922565818d54aab55ad4921"
server_key_iv: "cf0fa37499cb9aef02d5d9d596245859"
server_key_version: "encryptionkey@0"
I20260806 10:48:11.634979 24874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "50118dab0f804596a404eb955da7bac0"
format_stamp: "Formatted at 2026-08-06 10:48:11 on dist-test-slave-0b58"
server_key: "78c115b26922565818d54aab55ad4921"
server_key_iv: "cf0fa37499cb9aef02d5d9d596245859"
server_key_version: "encryptionkey@0"
I20260806 10:48:11.640079 24874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:48:11.643392 24889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:11.644745 24874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:11.644922 24874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "50118dab0f804596a404eb955da7bac0"
format_stamp: "Formatted at 2026-08-06 10:48:11 on dist-test-slave-0b58"
server_key: "78c115b26922565818d54aab55ad4921"
server_key_iv: "cf0fa37499cb9aef02d5d9d596245859"
server_key_version: "encryptionkey@0"
I20260806 10:48:11.645113 24874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:11.662206 24874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:11.665958 24874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:11.666258 24874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:11.684840 24874 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35175
I20260806 10:48:11.684878 24940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35175 every 8 connection(s)
I20260806 10:48:11.686124 24874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:48:11.689721 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:11.690295 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 24874
I20260806 10:48:11.690524 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:48:11.691195 16468 external_mini_cluster.cc:1428] Setting key 52eb3f9843087c7232ff60817f87630b
I20260806 10:48:11.704061 24941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Bootstrap starting.
Aug 06 10:48:11 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:11.707180 24941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:11.708570 24941 log.cc:824] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:11.711760 24941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: No bootstrap required, opened a new log
I20260806 10:48:11.715345 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:11.693215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.16.21.62:35175)
0806 10:48:11.694127 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:11.694143 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:11.694202 (+ 59us) server_negotiation.cc:407] Connection header received
0806 10:48:11.696513 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:11.696541 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:11.697015 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:11.697454 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:11.698786 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:11.700817 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:11.702052 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:11.703072 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:11.703254 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:11.707044 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:11.707060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:11.707079 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:11.707125 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:11.710924 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:11.711565 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:11.711573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:11.711581 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:11.711800 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:11.712184 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:11.712191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:11.712200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:11.712910 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:11.713202 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:11.713566 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 10:48:11.714005 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":404,"threads_started":1}
I20260806 10:48:11.722009 24941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:11.722335 24941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:11.722437 24941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50118dab0f804596a404eb955da7bac0, State: Initialized, Role: FOLLOWER
I20260806 10:48:11.723059 24941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:11.723364 24941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:48:11.723518 24941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:48:11.723681 24941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:11.725946 24941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:11.726557 24941 leader_election.cc:302] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50118dab0f804596a404eb955da7bac0; no voters:
I20260806 10:48:11.727159 24941 leader_election.cc:288] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:48:11.727298 24946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:11.728061 24946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 LEADER]: Becoming Leader. State: Replica: 50118dab0f804596a404eb955da7bac0, State: Running, Role: LEADER
I20260806 10:48:11.728607 24946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:11.729225 24941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:48:11.731187 24947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50118dab0f804596a404eb955da7bac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } } }
I20260806 10:48:11.731519 24947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:11.732301 24948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50118dab0f804596a404eb955da7bac0. Latest consensus state: current_term: 1 leader_uuid: "50118dab0f804596a404eb955da7bac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } } }
I20260806 10:48:11.732775 24948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:11.733477 24953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:11.737941 24953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:11.745908 24953 catalog_manager.cc:1347] Generated new cluster ID: 85c09b1705e749b690839e5f3dfb939e
I20260806 10:48:11.746014 24953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:11.766477 24953 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:48:11.768211 24953 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:11.775374 24953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Generated new TSK 0
I20260806 10:48:11.776300 24953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:48:11.843082 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35175
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:12.178472 24967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:12.178726 24967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:12.178887 24967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:12.185822 24967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:12.185911 24967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:12.186096 24967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:48:12.195945 24967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:12.197084 24967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:12.198458 24967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:12.210945 24972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:12.211030 24975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:12.210990 24973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:12.212198 24967 server_base.cc:1034] running on GCE node
I20260806 10:48:12.212858 24967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:12.213634 24967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:12.214960 24967 hybrid_clock.cc:648] HybridClock initialized: now 1786013292215216 us; error 343 us; skew 500 ppm
I20260806 10:48:12.215224 24967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:12 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:12.218216 24967 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:48:12.219506 24967 webserver.cc:460] Webserver started at http://127.16.21.1:35513/ using document root <none> and password file <none>
I20260806 10:48:12.220127 24967 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:12.220315 24967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:12.220630 24967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:12.223806 24967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "3a2f216d3dac47248f56395e221f898d"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "99e9b60d0a20ae8d8929ebc9c59b373d"
server_key_iv: "21b63c78f84cb3f8639894cc7cad4a68"
server_key_version: "encryptionkey@0"
I20260806 10:48:12.224891 24967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "3a2f216d3dac47248f56395e221f898d"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "99e9b60d0a20ae8d8929ebc9c59b373d"
server_key_iv: "21b63c78f84cb3f8639894cc7cad4a68"
server_key_version: "encryptionkey@0"
I20260806 10:48:12.231879 24967 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260806 10:48:12.235543 24982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:12.236872 24967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:12.236992 24967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "3a2f216d3dac47248f56395e221f898d"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "99e9b60d0a20ae8d8929ebc9c59b373d"
server_key_iv: "21b63c78f84cb3f8639894cc7cad4a68"
server_key_version: "encryptionkey@0"
I20260806 10:48:12.237138 24967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:12.250423 24967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:12.253940 24967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:12.254156 24967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:12.254947 24967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:12.256086 24967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:12.256156 24967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:12.256240 24967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:12.256368 24967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 10:48:12.292717 24967 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:46717
I20260806 10:48:12.292800 25096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:46717 every 8 connection(s)
I20260806 10:48:12.293973 24967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:48:12.299721 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 24967
I20260806 10:48:12.299958 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:48:12.300848 16468 external_mini_cluster.cc:1428] Setting key b3c39c27200a84a7a303c1e3efb11d17
Aug 06 10:48:12 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:12.314903 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:12.296420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:38871 (local address 127.16.21.62:35175)
0806 10:48:12.296854 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:12.296859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:12.297855 (+ 996us) server_negotiation.cc:407] Connection header received
0806 10:48:12.299721 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:12.299727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:12.299823 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:12.299947 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:12.301907 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.302786 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.304394 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.304795 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.304947 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:12.308861 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:12.308866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:12.308869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:12.308899 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:12.311517 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:12.312462 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:12.312466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:12.312468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:12.312587 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:12.313071 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:12.313074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:12.313076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:12.313316 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:12.313426 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:12.314231 (+ 805us) server_negotiation.cc:299] Negotiation successful
0806 10:48:12.314422 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":206,"threads_started":1}
I20260806 10:48:12.315629 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:12.296947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:38871)
0806 10:48:12.297705 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:12.297742 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:12.299451 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:12.300279 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:12.300294 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:12.301046 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:12.301747 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:12.301766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.302972 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.302976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:12.304210 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:12.304223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.305036 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.305045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:12.305278 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:12.305933 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:12.305954 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:12.308627 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:12.311668 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:12.311679 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:12.311696 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:12.312344 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:12.312690 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:12.312697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:12.312703 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:12.312959 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:12.313438 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:12.313449 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:12.313859 (+ 410us) client_negotiation.cc:769] Sending connection context
0806 10:48:12.314135 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 10:48:12.314600 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":300,"threads_started":1}
I20260806 10:48:12.317067 25097 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35175
I20260806 10:48:12.317353 25097 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:12.318014 25097 heartbeater.cc:511] Master 127.16.21.62:35175 requested a full tablet report, sending...
I20260806 10:48:12.319940 24906 ts_manager.cc:194] Registered new tserver with Master: 3a2f216d3dac47248f56395e221f898d (127.16.21.1:46717)
I20260806 10:48:12.321424 24906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:38871
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:48:12.358675 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35175
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:12.679123 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:12.679432 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:12.679600 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:12.686833 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:12.686969 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:12.687142 25103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:48:12.696480 25103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:12.697602 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:12.699277 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:12.711853 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:12.711874 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:12.712121 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:12.712949 25103 server_base.cc:1034] running on GCE node
I20260806 10:48:12.713488 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:12.714352 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:12.715575 25103 hybrid_clock.cc:648] HybridClock initialized: now 1786013292715535 us; error 50 us; skew 500 ppm
I20260806 10:48:12.715876 25103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:12 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:12.718994 25103 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:48:12.720333 25103 webserver.cc:460] Webserver started at http://127.16.21.2:35441/ using document root <none> and password file <none>
I20260806 10:48:12.720934 25103 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:12.721031 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:12.721287 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:12.723897 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "30811a04a28c443387ba50f206fbad30"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "4841d8de47acf88a568390956c99b3b4"
server_key_iv: "1db61722597c4fcb9c16e08f6643f501"
server_key_version: "encryptionkey@0"
I20260806 10:48:12.724637 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "30811a04a28c443387ba50f206fbad30"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "4841d8de47acf88a568390956c99b3b4"
server_key_iv: "1db61722597c4fcb9c16e08f6643f501"
server_key_version: "encryptionkey@0"
I20260806 10:48:12.729647 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:48:12.733013 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:12.734243 25103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:48:12.734464 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "30811a04a28c443387ba50f206fbad30"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "4841d8de47acf88a568390956c99b3b4"
server_key_iv: "1db61722597c4fcb9c16e08f6643f501"
server_key_version: "encryptionkey@0"
I20260806 10:48:12.734653 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:12.759482 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:12.763175 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:12.763484 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:12.764443 25103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:12.765715 25103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:12.765818 25103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:12.765962 25103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:12.766036 25103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:12.799839 25103 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:45827
I20260806 10:48:12.799885 25231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:45827 every 8 connection(s)
I20260806 10:48:12.801169 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:48:12.810144 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25103
I20260806 10:48:12.810329 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:48:12.810779 16468 external_mini_cluster.cc:1428] Setting key 626bf2f46d86d2a07ca9babf46b3999e
Aug 06 10:48:12 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:12.822556 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:12.803624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:49779 (local address 127.16.21.62:35175)
0806 10:48:12.803847 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:12.803853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:12.805048 (+ 1195us) server_negotiation.cc:407] Connection header received
0806 10:48:12.807038 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:12.807043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:12.807113 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:12.807200 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:12.808661 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.809405 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.811046 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.811630 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.811773 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:12.815780 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:12.815788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:12.815795 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:12.815841 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:12.819326 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:12.820020 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:12.820027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:12.820033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:12.820192 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:12.820660 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:12.820667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:12.820672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:12.821042 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:12.821200 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:12.821845 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 10:48:12.822097 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 10:48:12.823200 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:12.804123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:49779)
0806 10:48:12.804886 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:12.804936 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:12.806798 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:12.807404 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:12.807412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:12.807898 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:12.808512 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:12.808532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.809537 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.809541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:12.810909 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:12.810921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:12.811868 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:12.811876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:12.812104 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:12.812901 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:12.812923 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:12.815547 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:12.819461 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:12.819468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:12.819479 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:12.819886 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:12.820342 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:12.820345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:12.820348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:12.820537 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:12.821164 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:12.821170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:12.821471 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 10:48:12.821690 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 10:48:12.822078 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":346,"threads_started":1}
I20260806 10:48:12.824697 25232 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35175
I20260806 10:48:12.825035 25232 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:12.825873 25232 heartbeater.cc:511] Master 127.16.21.62:35175 requested a full tablet report, sending...
I20260806 10:48:12.827477 24906 ts_manager.cc:194] Registered new tserver with Master: 30811a04a28c443387ba50f206fbad30 (127.16.21.2:45827)
I20260806 10:48:12.828753 24906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:49779
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:48:12.872843 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35175
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:13.196954 25237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:13.197207 25237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:13.197307 25237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:13.204245 25237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:13.204406 25237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:13.204546 25237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:48:13.213893 25237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:35175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:13.215143 25237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:13.216365 25237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:13.229176 25245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:13.229252 25242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:13.229189 25243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:13.230115 25237 server_base.cc:1034] running on GCE node
I20260806 10:48:13.230718 25237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:13.231606 25237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:13.232898 25237 hybrid_clock.cc:648] HybridClock initialized: now 1786013293232823 us; error 81 us; skew 500 ppm
I20260806 10:48:13.233170 25237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:13 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013293, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:13.236209 25237 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:48:13.237655 25237 webserver.cc:460] Webserver started at http://127.16.21.3:44265/ using document root <none> and password file <none>
I20260806 10:48:13.238377 25237 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:13.238472 25237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:13.238737 25237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:13.241604 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "03af5a5db43c4acdac58b9ea2fd71b65"
format_stamp: "Formatted at 2026-08-06 10:48:13 on dist-test-slave-0b58"
server_key: "30ffebd81bdd7bf8d5f5fe2423a529b6"
server_key_iv: "28f791e89e0f48b62449040b7a18c1f7"
server_key_version: "encryptionkey@0"
I20260806 10:48:13.242333 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "03af5a5db43c4acdac58b9ea2fd71b65"
format_stamp: "Formatted at 2026-08-06 10:48:13 on dist-test-slave-0b58"
server_key: "30ffebd81bdd7bf8d5f5fe2423a529b6"
server_key_iv: "28f791e89e0f48b62449040b7a18c1f7"
server_key_version: "encryptionkey@0"
I20260806 10:48:13.247134 25237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:48:13.250505 25252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:13.251834 25237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:48:13.251965 25237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "03af5a5db43c4acdac58b9ea2fd71b65"
format_stamp: "Formatted at 2026-08-06 10:48:13 on dist-test-slave-0b58"
server_key: "30ffebd81bdd7bf8d5f5fe2423a529b6"
server_key_iv: "28f791e89e0f48b62449040b7a18c1f7"
server_key_version: "encryptionkey@0"
I20260806 10:48:13.252127 25237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:13.265166 25237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:13.268904 25237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:13.269170 25237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:13.269992 25237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:13.271184 25237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:13.271256 25237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:13.271337 25237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:13.271384 25237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:13.304492 25237 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41417
I20260806 10:48:13.304515 25365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41417 every 8 connection(s)
I20260806 10:48:13.305642 25237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:48:13.315119 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25237
I20260806 10:48:13.315338 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:48:13.315889 16468 external_mini_cluster.cc:1428] Setting key 1ad5c1f231f751d2ffdfd40e098f039c
Aug 06 10:48:13 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013293, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:13.324043 25097 heartbeater.cc:503] Master 127.16.21.62:35175 was elected leader, sending a full tablet report...
I20260806 10:48:13.324586 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:13.308105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:55437 (local address 127.16.21.62:35175)
0806 10:48:13.308311 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:13.308315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:13.309376 (+ 1061us) server_negotiation.cc:407] Connection header received
0806 10:48:13.311055 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:13.311060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:13.311164 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:13.311250 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:13.312646 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.313367 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.314664 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.315028 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.315110 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:13.319306 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:13.319311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:13.319315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:13.319346 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:13.321746 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.322419 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.322423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.322425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.322543 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.322970 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.322974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.322976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.323228 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:13.323375 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:13.324001 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 10:48:13.324203 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:48:13.325529 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:13.308552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:55437)
0806 10:48:13.309213 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:13.309246 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:13.310827 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:13.311432 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:13.311439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:13.311913 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:13.312505 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:13.312523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.313479 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.313482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:13.314555 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:13.314565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.315263 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.315273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:13.315563 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:13.316296 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:13.316328 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:13.319099 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:13.321892 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:13.321899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:13.321911 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:13.322293 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:13.322660 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:13.322665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:13.322668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:13.322848 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:13.323348 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:13.323354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:13.323628 (+ 274us) client_negotiation.cc:769] Sending connection context
0806 10:48:13.323889 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 10:48:13.324350 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
I20260806 10:48:13.326797 25366 heartbeater.cc:347] Connected to a master server at 127.16.21.62:35175
I20260806 10:48:13.327073 25366 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:13.327742 25366 heartbeater.cc:511] Master 127.16.21.62:35175 requested a full tablet report, sending...
I20260806 10:48:13.329209 24906 ts_manager.cc:194] Registered new tserver with Master: 03af5a5db43c4acdac58b9ea2fd71b65 (127.16.21.3:41417)
I20260806 10:48:13.330312 24906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:55437
I20260806 10:48:13.331547 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab.
Entry for principal oryx/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab.
WARNING: no policy specified for oryx/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab.
Entry for principal oryx/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab.
WARNING: no policy specified for oryx/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab.
Entry for principal oryx/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab.
I20260806 10:48:13.486646 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 24967
Aug 06 10:48:13 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:48:13.502439 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:13.487778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.16.21.1:46717)
0806 10:48:13.488499 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:13.488506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:13.488532 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:48:13.488595 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:13.488601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:13.488812 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:13.488947 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:13.490234 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.491220 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.492433 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.492963 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.493056 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:13.496983 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:13.496994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:13.497008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:13.497041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:13.499601 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.500307 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.500323 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.500325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.500453 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.500869 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.500873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.500875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.501173 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:13.501499 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:13.501820 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 10:48:13.502084 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":353,"threads_started":1}
I20260806 10:48:13.503598 25092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:13.587144 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 24967
I20260806 10:48:13.606154 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25103
Aug 06 10:48:13 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:48:13.621357 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:13.607089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.16.21.2:45827)
0806 10:48:13.607629 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:13.607636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:13.607661 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:48:13.607730 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:13.607736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:13.607942 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:13.608071 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:13.609298 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.610276 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.611528 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.612050 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.612143 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:13.616203 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:13.616215 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:13.616227 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:13.616316 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:13.618971 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.619578 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.619584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.619587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.619703 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.620005 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.620008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.620010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.620399 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:13.620648 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:13.620862 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 10:48:13.621068 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
I20260806 10:48:13.622597 25227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:13.709805 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25103
I20260806 10:48:13.728890 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25237
Aug 06 10:48:13 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:48:13.746019 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:13.729846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.16.21.3:41417)
0806 10:48:13.730593 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:13.730599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:13.730630 (+ 31us) server_negotiation.cc:407] Connection header received
0806 10:48:13.730714 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:13.730723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:13.730974 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:13.731101 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:13.732467 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.733507 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.734777 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:13.735345 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:13.735438 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:13.740074 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:13.740086 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:13.740099 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:13.740130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:13.743469 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.744105 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.744112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.744114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.744228 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:13.744610 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:13.744613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:13.744615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:13.744943 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:13.745184 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:13.745418 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 10:48:13.745609 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":303,"threads_started":1}
I20260806 10:48:13.747311 25361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:13.830080 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25237
I20260806 10:48:13.850104 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 24874
I20260806 10:48:13.851301 24936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:13.955885 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 24874
I20260806 10:48:13.980806 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35175
--webserver_interface=127.16.21.62
--webserver_port=45149
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:14.311028 25383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:14.311295 25383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:14.311453 25383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:14.318292 25383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:14.318404 25383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:14.318449 25383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:14.318526 25383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:14.318591 25383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:14.327818 25383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=45149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:14.328984 25383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:14.330240 25383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:14.342319 25390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:14.342300 25392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:14.342383 25389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:14.343227 25383 server_base.cc:1034] running on GCE node
I20260806 10:48:14.343933 25383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:14.345340 25383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:14.346575 25383 hybrid_clock.cc:648] HybridClock initialized: now 1786013294346527 us; error 54 us; skew 500 ppm
I20260806 10:48:14.346927 25383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013294, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:14.350096 25383 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:48:14.351627 25383 webserver.cc:460] Webserver started at http://127.16.21.62:45149/ using document root <none> and password file <none>
I20260806 10:48:14.352367 25383 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:14.352526 25383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:14.357748 25383 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 10:48:14.360463 25399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:14.361687 25383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:48:14.361864 25383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "50118dab0f804596a404eb955da7bac0"
format_stamp: "Formatted at 2026-08-06 10:48:11 on dist-test-slave-0b58"
server_key: "78c115b26922565818d54aab55ad4921"
server_key_iv: "cf0fa37499cb9aef02d5d9d596245859"
server_key_version: "encryptionkey@0"
I20260806 10:48:14.362823 25383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:14.385747 25383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:14.389498 25383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:14.389767 25383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:14.409948 25383 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35175
I20260806 10:48:14.409972 25450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35175 every 8 connection(s)
I20260806 10:48:14.411338 25383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:48:14.418042 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25383
I20260806 10:48:14.418490 25451 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:48:14.419059 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:46717
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:35175
--webserver_port=35513
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:48:14.423398 25451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Bootstrap starting.
I20260806 10:48:14.445497 25451 log.cc:824] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:14.453941 25451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:48:14.454474 25451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Bootstrap complete.
I20260806 10:48:14.463198 25451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:14.463922 25451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50118dab0f804596a404eb955da7bac0, State: Initialized, Role: FOLLOWER
I20260806 10:48:14.464525 25451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:14.464789 25451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:48:14.464938 25451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:48:14.465129 25451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:48:14.469201 25451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:14.469696 25451 leader_election.cc:302] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50118dab0f804596a404eb955da7bac0; no voters:
I20260806 10:48:14.470345 25451 leader_election.cc:288] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:48:14.470472 25455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:48:14.471705 25455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [term 2 LEADER]: Becoming Leader. State: Replica: 50118dab0f804596a404eb955da7bac0, State: Running, Role: LEADER
I20260806 10:48:14.472340 25455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } }
I20260806 10:48:14.472538 25451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:48:14.475180 25457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50118dab0f804596a404eb955da7bac0. Latest consensus state: current_term: 2 leader_uuid: "50118dab0f804596a404eb955da7bac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } } }
I20260806 10:48:14.475610 25457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:14.475907 25456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "50118dab0f804596a404eb955da7bac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50118dab0f804596a404eb955da7bac0" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35175 } } }
I20260806 10:48:14.476361 25456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:14.487279 25464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:14.493242 25464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:14.494534 25464 catalog_manager.cc:1259] Loaded cluster ID: 85c09b1705e749b690839e5f3dfb939e
I20260806 10:48:14.494619 25464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:14.498449 25464 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:14.499877 25472 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 10:48:14.499907 25464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 50118dab0f804596a404eb955da7bac0: Loaded TSK: 0
I20260806 10:48:14.501659 25464 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:48:14.765101 25453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:14.765399 25453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:14.765628 25453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:14.773249 25453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:14.773413 25453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:14.773600 25453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:48:14.783340 25453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.1.keytab
--principal=oryx/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:46717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=35513
--tserver_master_addrs=127.16.21.62:35175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:14.784669 25453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:14.785917 25453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:14.800290 25478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:14.800349 25477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:14.800325 25480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:14.801700 25453 server_base.cc:1034] running on GCE node
I20260806 10:48:14.802356 25453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:14.803300 25453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:14.804529 25453 hybrid_clock.cc:648] HybridClock initialized: now 1786013294804477 us; error 44 us; skew 500 ppm
I20260806 10:48:14.804891 25453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013294, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:14.808089 25453 init.cc:377] Logged in from keytab as oryx/127.16.21.1@KRBTEST.COM (short username oryx)
I20260806 10:48:14.809690 25453 webserver.cc:460] Webserver started at http://127.16.21.1:35513/ using document root <none> and password file <none>
I20260806 10:48:14.810374 25453 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:14.810513 25453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:14.815799 25453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 10:48:14.818519 25487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:14.819872 25453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:48:14.820044 25453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "3a2f216d3dac47248f56395e221f898d"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "99e9b60d0a20ae8d8929ebc9c59b373d"
server_key_iv: "21b63c78f84cb3f8639894cc7cad4a68"
server_key_version: "encryptionkey@0"
I20260806 10:48:14.821063 25453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:14.838152 25453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:14.842387 25453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:14.842672 25453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:14.843657 25453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:14.845005 25453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:14.845113 25453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:14.845249 25453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:14.845346 25453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:14.879405 25453 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:46717
I20260806 10:48:14.879495 25600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:46717 every 8 connection(s)
I20260806 10:48:14.880911 25453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:48:14.886775 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25453
I20260806 10:48:14.887411 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:45827
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:35175
--webserver_port=35441
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:14.899106 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:14.883930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:60839)
0806 10:48:14.884811 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:14.884857 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:14.886587 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:14.887989 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:14.888002 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:14.888652 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:14.889175 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:14.889194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.890787 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.890791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:14.891997 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:14.892011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.893148 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.893156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:14.893418 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:14.894101 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:14.894132 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:14.897836 (+ 3704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":388,"threads_started":1}
W20260806 10:48:14.899253 25603 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:14.883851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:60839 (local address 127.16.21.62:35175)
0806 10:48:14.884674 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:14.884689 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:14.884982 (+ 293us) server_negotiation.cc:407] Connection header received
0806 10:48:14.887116 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:14.887128 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:14.887440 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:14.887802 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:14.889400 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.890608 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.892145 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.892865 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.893057 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:14.898230 (+ 5173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:60839: BlockingRecv error: recv got EOF from 127.16.21.1:60839 (error 108)
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":311,"threads_started":1}
W20260806 10:48:14.899861 25601 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:35175 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.21.62:35175: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:14.908684 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:14.900506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:47853)
0806 10:48:14.900709 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:14.900729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:14.900824 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:14.901179 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:14.901182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:14.901470 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:14.901596 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:14.901603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.903067 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.903071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:14.903930 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:14.903940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.904984 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.904988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:14.905108 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:14.905675 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:14.905692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:14.908496 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 10:48:14.909013 25603 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:14.900676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:47853 (local address 127.16.21.62:35175)
0806 10:48:14.900875 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:14.900883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:14.900904 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:48:14.900958 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:14.900965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:14.901044 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:14.901152 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:14.901730 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.902929 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.904072 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.904838 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.905004 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:14.908751 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:47853: BlockingRecv error: recv got EOF from 127.16.21.1:47853 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:14 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:14.918568 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:14.909388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:54689)
0806 10:48:14.909593 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:14.909616 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:14.909727 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:14.910056 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:14.910064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:14.910472 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:14.910685 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:14.910699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.911643 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.911651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:14.913438 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:14.913457 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.914149 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.914158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:14.914445 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:14.915040 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:14.915069 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:14.918255 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 10:48:14.918726 25603 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:14.909471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:54689 (local address 127.16.21.62:35175)
0806 10:48:14.909632 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:14.909637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:14.909655 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:48:14.909853 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:14.909856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:14.909921 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:14.909995 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:14.910820 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.911494 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.913583 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:14.913981 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:14.914079 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:14.918451 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:54689: BlockingRecv error: recv got EOF from 127.16.21.1:54689 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 10:48:14.918893 25601 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:48:15.226019 25605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:15.226320 25605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:15.226415 25605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:15.233538 25605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:15.233636 25605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:15.233754 25605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:48:15.243182 25605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.2.keytab
--principal=oryx/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:45827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=35441
--tserver_master_addrs=127.16.21.62:35175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:15.244371 25605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:15.245599 25605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:15.258224 25610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:15.258471 25611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:15.258574 25613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:15.259330 25605 server_base.cc:1034] running on GCE node
I20260806 10:48:15.259949 25605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:15.260911 25605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:15.262111 25605 hybrid_clock.cc:648] HybridClock initialized: now 1786013295262071 us; error 30 us; skew 500 ppm
I20260806 10:48:15.262458 25605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013295, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:15.265558 25605 init.cc:377] Logged in from keytab as oryx/127.16.21.2@KRBTEST.COM (short username oryx)
I20260806 10:48:15.267213 25605 webserver.cc:460] Webserver started at http://127.16.21.2:35441/ using document root <none> and password file <none>
I20260806 10:48:15.267905 25605 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:15.268062 25605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:15.273427 25605 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:48:15.276005 25620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:15.277448 25605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:48:15.277614 25605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "30811a04a28c443387ba50f206fbad30"
format_stamp: "Formatted at 2026-08-06 10:48:12 on dist-test-slave-0b58"
server_key: "4841d8de47acf88a568390956c99b3b4"
server_key_iv: "1db61722597c4fcb9c16e08f6643f501"
server_key_version: "encryptionkey@0"
I20260806 10:48:15.278563 25605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:15.294669 25605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:15.298377 25605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:15.298648 25605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:15.299561 25605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:15.300915 25605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:15.301038 25605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:15.301177 25605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:15.301271 25605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:15.336083 25605 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:45827
I20260806 10:48:15.336171 25734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:45827 every 8 connection(s)
I20260806 10:48:15.337574 25605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:48:15.339159 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25605
I20260806 10:48:15.339780 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:41417
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:35175
--webserver_port=44265
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:48:15.357041 25603 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.342047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:50841 (local address 127.16.21.62:35175)
0806 10:48:15.342291 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.342300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.343442 (+ 1142us) server_negotiation.cc:407] Connection header received
0806 10:48:15.345511 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.345519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.345614 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.345764 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.347213 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.348381 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.349918 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.350635 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.350787 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.356844 (+ 6057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:50841: BlockingRecv error: recv got EOF from 127.16.21.2:50841 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:48:15.358011 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.342468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:50841)
0806 10:48:15.343279 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.343322 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.345142 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.345954 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.345961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.346480 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.347027 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.347044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.348545 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.348548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.349771 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.349783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.350851 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.350855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.351051 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.351713 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.351734 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.356682 (+ 4948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":368,"threads_started":1}
W20260806 10:48:15.358633 25735 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:35175 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.21.62:35175: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:15.367566 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.359246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:51513)
0806 10:48:15.359570 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.359585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.359673 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.359978 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.359981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.360314 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.360450 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.360458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.361844 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.361847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.362684 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.362694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.363716 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.363720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.363844 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.364449 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.364465 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.367391 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 10:48:15.368119 25603 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.359352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:51513 (local address 127.16.21.62:35175)
0806 10:48:15.359502 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.359506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.359665 (+ 159us) server_negotiation.cc:407] Connection header received
0806 10:48:15.359762 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.359770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.359853 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.359953 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.360590 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.361704 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.362844 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.363576 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.363737 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.367807 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:51513: BlockingRecv error: recv got EOF from 127.16.21.2:51513 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:15.380007 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.368192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:53387)
0806 10:48:15.368712 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.368736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.368875 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.369248 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.369255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.369761 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.370018 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.370032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.372486 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.372494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.374095 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.374114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.375127 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.375135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.375384 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.376340 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.376362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.379723 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239}
W20260806 10:48:15.380409 25603 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.368292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:53387 (local address 127.16.21.62:35175)
0806 10:48:15.368534 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.368542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.368837 (+ 295us) server_negotiation.cc:407] Connection header received
0806 10:48:15.369003 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.369010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.369093 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.369202 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.370195 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.372299 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.374238 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.374955 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.375107 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.380054 (+ 4947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:53387: BlockingRecv error: recv got EOF from 127.16.21.2:53387 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"threads_started":1}
W20260806 10:48:15.381696 25735 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:48:15.696099 25738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:15.696461 25738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:15.696641 25738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:15.703554 25738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:15.703687 25738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:15.703876 25738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:48:15.713116 25738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/krb5kdc/oryx_127.16.21.3.keytab
--principal=oryx/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:41417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=44265
--tserver_master_addrs=127.16.21.62:35175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:15.714313 25738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:15.715546 25738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:15.728199 25744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:15.728154 25745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:15.728291 25747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:15.729312 25738 server_base.cc:1034] running on GCE node
I20260806 10:48:15.729933 25738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:15.730773 25738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:15.731999 25738 hybrid_clock.cc:648] HybridClock initialized: now 1786013295731962 us; error 60 us; skew 500 ppm
I20260806 10:48:15.732368 25738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013295, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:15.735874 25738 init.cc:377] Logged in from keytab as oryx/127.16.21.3@KRBTEST.COM (short username oryx)
I20260806 10:48:15.737360 25738 webserver.cc:460] Webserver started at http://127.16.21.3:44265/ using document root <none> and password file <none>
I20260806 10:48:15.738013 25738 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:15.738170 25738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:15.743168 25738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 10:48:15.745781 25754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:15.746963 25738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 10:48:15.747128 25738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "03af5a5db43c4acdac58b9ea2fd71b65"
format_stamp: "Formatted at 2026-08-06 10:48:13 on dist-test-slave-0b58"
server_key: "30ffebd81bdd7bf8d5f5fe2423a529b6"
server_key_iv: "28f791e89e0f48b62449040b7a18c1f7"
server_key_version: "encryptionkey@0"
I20260806 10:48:15.748055 25738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:15.771377 25738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:15.775584 25738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:15.775866 25738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:15.776909 25738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:15.778268 25738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:15.778409 25738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:15.778532 25738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:15.778623 25738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:15.812354 25738 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41417
I20260806 10:48:15.812441 25868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41417 every 8 connection(s)
I20260806 10:48:15.813745 25738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:48:15.816731 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25738
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:48:15.831339 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.816319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:44175 (local address 127.16.21.62:35175)
0806 10:48:15.816538 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.816544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.817897 (+ 1353us) server_negotiation.cc:407] Connection header received
0806 10:48:15.819910 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.819916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.820007 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.820130 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.821894 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.822918 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.824878 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.825469 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.825666 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.831039 (+ 5373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:44175: BlockingRecv error: recv got EOF from 127.16.21.3:44175 (error 108)
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":85}
I20260806 10:48:15.831970 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.816864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:44175)
0806 10:48:15.817737 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.817783 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.819646 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.820449 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.820457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.821156 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.821699 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.821718 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.823081 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.823087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.824727 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.824743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.825782 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.825790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.826092 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.826667 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.826705 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.830813 (+ 4108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":370,"threads_started":1}
W20260806 10:48:15.832671 25869 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:35175 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.21.62:35175: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
I20260806 10:48:15.836170 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.817834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.16.21.62:35175)
0806 10:48:15.818037 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.818046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.820562 (+ 2516us) server_negotiation.cc:407] Connection header received
0806 10:48:15.820671 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.820676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.820752 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.820871 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.822443 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.823396 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.824767 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.825438 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.825579 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.828902 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:15.828910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:15.828924 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:15.828959 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:15.832085 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:15.833551 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:15.833559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:15.833567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:15.833797 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:15.834212 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:15.834216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:15.834218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:15.834776 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:15.835097 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:15.835554 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 10:48:15.835903 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":1280}
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:15.841889 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.833179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:36469)
0806 10:48:15.833399 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.833419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.833522 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.833901 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.833905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.834294 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.834426 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.834434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.835833 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.835837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.836883 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.836896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.837698 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.837704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.837884 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.838673 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.838689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.841606 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 10:48:15.842265 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.833320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:36469 (local address 127.16.21.62:35175)
0806 10:48:15.833493 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.833500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.833518 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:48:15.833655 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.833662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.833734 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.833845 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.834561 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.835682 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.837023 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.837548 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.837644 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.841793 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:36469: BlockingRecv error: recv got EOF from 127.16.21.3:36469 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:15.849521 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.842483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:45321)
0806 10:48:15.842631 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.842646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.842716 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.843181 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.843185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.843454 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.843609 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.843617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.844645 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.844658 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.845539 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.845550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.846227 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.846233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.846395 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.846846 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.846861 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.849325 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:48:15.849717 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.842756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:45321 (local address 127.16.21.62:35175)
0806 10:48:15.842948 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.842952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.842964 (+ 12us) server_negotiation.cc:407] Connection header received
0806 10:48:15.842998 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.843001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.843057 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.843156 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.843753 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.844502 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.845675 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.846071 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.846207 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.849481 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:45321: BlockingRecv error: recv got EOF from 127.16.21.3:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260806 10:48:15.849792 25869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:15 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:15.928515 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:15.919630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:37415)
0806 10:48:15.920109 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:15.920124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:15.920236 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:15.920568 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:15.920572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:15.921027 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:15.921324 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.921337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.922786 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.922790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.923740 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:15.923751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.924493 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.924499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:15.924668 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.925314 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:15.925339 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:15.928152 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
W20260806 10:48:15.928624 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:15.919689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:37415 (local address 127.16.21.62:35175)
0806 10:48:15.919928 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:15.919935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:15.920208 (+ 273us) server_negotiation.cc:407] Connection header received
0806 10:48:15.920367 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:15.920371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:15.920442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:15.920521 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:15.921489 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.922654 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.923869 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:15.924324 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:15.924434 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:15.928398 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:37415: BlockingRecv error: recv got EOF from 127.16.21.1:37415 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 10:48:16 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:16 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:16.391086 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:16.382288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:53983)
0806 10:48:16.382816 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:16.382831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:16.382981 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:16.383384 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:16.383388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:16.383869 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:16.384057 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:16.384067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.385136 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.385142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:16.386143 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:16.386155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.386960 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.386967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:16.387130 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:16.387690 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:16.387707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:16.390868 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
W20260806 10:48:16.391280 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:16.382332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:53983 (local address 127.16.21.62:35175)
0806 10:48:16.382528 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:16.382536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:16.382916 (+ 380us) server_negotiation.cc:407] Connection header received
0806 10:48:16.383144 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:16.383149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:16.383234 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:16.383337 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:16.384205 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.384999 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.386323 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.386773 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.386901 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:16.391050 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:53983: BlockingRecv error: recv got EOF from 127.16.21.2:53983 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 10:48:16 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:16 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:16.867619 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:16.850329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:53743)
0806 10:48:16.850847 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:16.850863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:16.851015 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:16.851506 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:16.851513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:16.852007 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:16.852199 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:16.852208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.853333 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.853340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:16.854329 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:16.854341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.855121 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.855127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:16.855325 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:16.864345 (+ 9020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:16.864364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:16.867264 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":272,"threads_started":1}
W20260806 10:48:16.867748 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:16.850435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:53743 (local address 127.16.21.62:35175)
0806 10:48:16.850659 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:16.850666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:16.850946 (+ 280us) server_negotiation.cc:407] Connection header received
0806 10:48:16.851134 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:16.851140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:16.851217 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:16.851405 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:16.852388 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.853193 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.854483 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.854935 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.855074 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:16.867483 (+ 12409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:53743: BlockingRecv error: recv got EOF from 127.16.21.3:53743 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 10:48:16 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:16 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:16.937705 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:16.929431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:54613)
0806 10:48:16.929950 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:16.929966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:16.930124 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:16.930481 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:16.930486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:16.930917 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:16.931061 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:16.931070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.932079 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.932085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:16.933019 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:16.933030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.933753 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.933766 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:16.933956 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:16.934488 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:16.934505 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:16.937326 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":273,"threads_started":1}
W20260806 10:48:16.937790 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:16.929482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:54613 (local address 127.16.21.62:35175)
0806 10:48:16.929727 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:16.929735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:16.930055 (+ 320us) server_negotiation.cc:407] Connection header received
0806 10:48:16.930269 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:16.930275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:16.930348 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:16.930448 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:16.931209 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.931936 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.933163 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:16.933580 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:16.933724 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:16.937584 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:54613: BlockingRecv error: recv got EOF from 127.16.21.1:54613 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 10:48:17 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:17 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:17.399988 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:17.391985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:39307)
0806 10:48:17.392556 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:17.392572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:17.392684 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:17.392935 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:17.392938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:17.393318 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:17.393458 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:17.393466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.394406 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.394409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:17.395313 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:17.395325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.395981 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.395985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:17.396090 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:17.396668 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:17.396692 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:17.399740 (+ 3048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
W20260806 10:48:17.400121 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:17.392056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:39307 (local address 127.16.21.62:35175)
0806 10:48:17.392405 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:17.392410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:17.392642 (+ 232us) server_negotiation.cc:407] Connection header received
0806 10:48:17.392791 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:17.392795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:17.392850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:17.392921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:17.393576 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.394277 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.395451 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.395857 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.395949 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:17.399925 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:39307: BlockingRecv error: recv got EOF from 127.16.21.2:39307 (error 108)
Metrics: {"server-negotiator.queue_time_us":220}
Aug 06 10:48:17 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:17 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:17.877148 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:17.868640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:59601)
0806 10:48:17.869179 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:17.869195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:17.869350 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:17.869706 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:17.869711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:17.870224 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:17.870378 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:17.870388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.871438 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.871444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:17.872390 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:17.872401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.873269 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.873275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:17.873500 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:17.874152 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:17.874170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:17.876834 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":280,"threads_started":1}
W20260806 10:48:17.877303 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:17.868827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:59601 (local address 127.16.21.62:35175)
0806 10:48:17.869044 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:17.869050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:17.869279 (+ 229us) server_negotiation.cc:407] Connection header received
0806 10:48:17.869478 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:17.869490 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:17.869564 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:17.869674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:17.870523 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.871287 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.872524 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.873095 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.873227 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:17.877019 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:59601: BlockingRecv error: recv got EOF from 127.16.21.3:59601 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 10:48:17 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:17 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:17.946962 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:17.938781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:50471)
0806 10:48:17.939385 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:17.939411 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:17.939527 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:17.939854 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:17.939857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:17.940188 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:17.940428 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:17.940438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.941459 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.941463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:17.942347 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:17.942358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.943029 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.943035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:17.943212 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:17.943860 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:17.943897 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:17.946651 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":332,"threads_started":1}
W20260806 10:48:17.947147 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:17.938843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:50471 (local address 127.16.21.62:35175)
0806 10:48:17.939001 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:17.939006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:17.939518 (+ 512us) server_negotiation.cc:407] Connection header received
0806 10:48:17.939630 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:17.939633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:17.939716 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:17.939802 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:17.940565 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.941312 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.942465 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:17.942882 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:17.943010 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:17.946873 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:50471: BlockingRecv error: recv got EOF from 127.16.21.1:50471 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 10:48:18 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:18 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:18.409242 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:18.401035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:45073)
0806 10:48:18.401725 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:18.401746 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:18.401892 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:18.402260 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:18.402264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:18.402593 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:18.402769 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:18.402778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.403831 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.403835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:18.404859 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:18.404872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.405531 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.405535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:18.405660 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:18.406332 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:18.406349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:18.408997 (+ 2648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":430,"threads_started":1}
W20260806 10:48:18.409389 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:18.401096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45073 (local address 127.16.21.62:35175)
0806 10:48:18.401323 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:18.401350 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:18.401838 (+ 488us) server_negotiation.cc:407] Connection header received
0806 10:48:18.402047 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:18.402052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:18.402125 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:18.402202 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:18.402899 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.403654 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.404997 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.405405 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.405496 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:18.409175 (+ 3679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:45073: BlockingRecv error: recv got EOF from 127.16.21.2:45073 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 10:48:18 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:18 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:18.887413 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:18.878081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:48291)
0806 10:48:18.878571 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:18.878586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:18.878740 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:18.879116 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:18.879120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:18.879542 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:18.879726 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:18.879738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.881198 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.881202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:18.882556 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:18.882569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.883527 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.883531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:18.883685 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:18.884457 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:18.884477 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:18.887156 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":272,"threads_started":1}
W20260806 10:48:18.887697 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:18.878173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:48291 (local address 127.16.21.62:35175)
0806 10:48:18.878407 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:18.878416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:18.878668 (+ 252us) server_negotiation.cc:407] Connection header received
0806 10:48:18.878858 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:18.878867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:18.878967 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:18.879063 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:18.879896 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.881077 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.882740 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.883392 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.883489 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:18.887351 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:48291: BlockingRecv error: recv got EOF from 127.16.21.3:48291 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 10:48:18 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:18 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:18.956543 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:18.947976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:45921)
0806 10:48:18.948518 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:18.948534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:18.948644 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:18.948947 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:18.948951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:18.949308 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:18.949456 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:18.949464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.950447 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.950452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:18.951543 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:18.951556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.952606 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.952612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:18.952798 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:18.953502 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:18.953543 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:18.956238 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":327,"threads_started":1}
W20260806 10:48:18.956718 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:18.948046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:45921 (local address 127.16.21.62:35175)
0806 10:48:18.948314 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:18.948324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:18.948609 (+ 285us) server_negotiation.cc:407] Connection header received
0806 10:48:18.948752 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:18.948756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:18.948825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:18.948903 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:18.949572 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.950316 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.951666 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:18.952397 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:18.952548 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:18.956485 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:45921: BlockingRecv error: recv got EOF from 127.16.21.1:45921 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 10:48:19 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:19 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:19.419178 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:19.410217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:49089)
0806 10:48:19.410682 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:19.410696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:19.410844 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:19.411157 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:19.411162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:19.411553 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:19.411704 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:19.411713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.412842 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.412849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:19.413849 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:19.413860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.414577 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.414583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:19.414782 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:19.415382 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:19.415399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:19.418897 (+ 3498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
W20260806 10:48:19.419288 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:19.410287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:49089 (local address 127.16.21.62:35175)
0806 10:48:19.410478 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:19.410486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:19.410783 (+ 297us) server_negotiation.cc:407] Connection header received
0806 10:48:19.410960 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:19.410969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:19.411027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:19.411124 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:19.411844 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.412698 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.413986 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.414439 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.414567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:19.419093 (+ 4526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:49089: BlockingRecv error: recv got EOF from 127.16.21.2:49089 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 10:48:19 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:19 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:19.896684 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:19.888285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:52319)
0806 10:48:19.888752 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:19.888767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:19.888932 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:19.889276 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:19.889280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:19.889681 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:19.889838 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:19.889847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.890875 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.890882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:19.891862 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:19.891873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.892664 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.892670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:19.892866 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:19.893507 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:19.893525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:19.896363 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
W20260806 10:48:19.896852 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:19.888358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:52319 (local address 127.16.21.62:35175)
0806 10:48:19.888570 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:19.888578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:19.888868 (+ 290us) server_negotiation.cc:407] Connection header received
0806 10:48:19.889050 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:19.889055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:19.889141 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:19.889237 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:19.890018 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.890743 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.892002 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.892519 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.892649 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:19.896558 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:52319: BlockingRecv error: recv got EOF from 127.16.21.3:52319 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 10:48:19 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:19 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:19.965804 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:19.957631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:48355)
0806 10:48:19.958148 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:19.958163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:19.958312 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:19.958653 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:19.958659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:19.959084 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:19.959253 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:19.959262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.960284 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.960292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:19.961236 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:19.961248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.961898 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.961905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:19.962071 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:19.962652 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:19.962670 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:19.965462 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":257,"threads_started":1}
W20260806 10:48:19.965955 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:19.957711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:48355 (local address 127.16.21.62:35175)
0806 10:48:19.957925 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:19.957933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:19.958239 (+ 306us) server_negotiation.cc:407] Connection header received
0806 10:48:19.958453 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:19.958456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:19.958515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:19.958610 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:19.959390 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.960121 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.961376 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:19.961750 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:19.961872 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:19.965685 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:48355: BlockingRecv error: recv got EOF from 127.16.21.1:48355 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 10:48:20 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:20 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:20.429584 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:20.420143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:60383)
0806 10:48:20.420739 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:20.420758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:20.420888 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:20.421915 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:20.421921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:20.422530 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:20.422726 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:20.422741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.424204 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.424208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:20.425150 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:20.425162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.425782 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.425786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:20.425895 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:20.426433 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:20.426449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:20.429243 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":280,"threads_started":1}
W20260806 10:48:20.429649 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:20.420304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:60383 (local address 127.16.21.62:35175)
0806 10:48:20.420958 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:20.420963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:20.420983 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:48:20.421035 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:20.421040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:20.421131 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:20.421841 (+ 710us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:20.422892 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.424064 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.425265 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.425668 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.425755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:20.429436 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:60383: BlockingRecv error: recv got EOF from 127.16.21.2:60383 (error 108)
Metrics: {"server-negotiator.queue_time_us":510}
Aug 06 10:48:20 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:20 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:20.906491 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:20.897691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:36153)
0806 10:48:20.898240 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:20.898256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:20.898407 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:20.898751 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:20.898755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:20.899200 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:20.899362 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:20.899371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.900654 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.900661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:20.901797 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:20.901810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.902536 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.902542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:20.902707 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:20.903262 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:20.903280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:20.906112 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":281,"threads_started":1}
W20260806 10:48:20.906567 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:20.897902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:36153 (local address 127.16.21.62:35175)
0806 10:48:20.898124 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:20.898131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:20.898340 (+ 209us) server_negotiation.cc:407] Connection header received
0806 10:48:20.898527 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:20.898536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:20.898610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:20.898711 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:20.899505 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.900508 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.901947 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.902386 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.902522 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:20.906317 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:36153: BlockingRecv error: recv got EOF from 127.16.21.3:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 10:48:20 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:20 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:20.974794 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:20.966681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:55679)
0806 10:48:20.967230 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:20.967245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:20.967360 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:20.967637 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:20.967640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:20.968010 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:20.968152 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:20.968159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.969106 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.969111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:20.970518 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:20.970544 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.971224 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.971228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:20.971350 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:20.971893 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:20.971911 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:20.974589 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":293,"threads_started":1}
W20260806 10:48:20.975031 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:20.966689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:55679 (local address 127.16.21.62:35175)
0806 10:48:20.966985 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:20.966994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:20.967330 (+ 336us) server_negotiation.cc:407] Connection header received
0806 10:48:20.967457 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:20.967460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:20.967519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:20.967596 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:20.968281 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.968953 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.970680 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:20.971092 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:20.971184 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:20.974775 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:55679: BlockingRecv error: recv got EOF from 127.16.21.1:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 10:48:21 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:21 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:21.439429 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:21.430559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:33575)
0806 10:48:21.431075 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:21.431091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:21.431242 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:21.431583 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:21.431588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:21.432021 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:21.432175 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:21.432183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.433318 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.433325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:21.434388 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:21.434401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.435242 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.435249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:21.435439 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:21.436154 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:21.436195 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:21.439221 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":302,"threads_started":1}
W20260806 10:48:21.439707 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:21.430592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:33575 (local address 127.16.21.62:35175)
0806 10:48:21.430790 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:21.430799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:21.431173 (+ 374us) server_negotiation.cc:407] Connection header received
0806 10:48:21.431372 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:21.431378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:21.431451 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:21.431548 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:21.432349 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.433066 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.434564 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.435067 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.435212 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:21.439416 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:33575: BlockingRecv error: recv got EOF from 127.16.21.2:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 10:48:21 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:21 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:21.919615 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:21.907541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:39519)
0806 10:48:21.908091 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:21.908108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:21.908365 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:21.908760 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:21.908766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:21.909294 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:21.909455 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:21.909465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.910613 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.910620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:21.911666 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:21.911680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.912656 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.912662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:21.912859 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:21.916615 (+ 3756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:21.916632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:21.919382 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":297,"threads_started":1}
W20260806 10:48:21.919800 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:21.907638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:39519 (local address 127.16.21.62:35175)
0806 10:48:21.907893 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:21.907903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:21.908198 (+ 295us) server_negotiation.cc:407] Connection header received
0806 10:48:21.908502 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:21.908513 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:21.908594 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:21.908735 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:21.909614 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.910463 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.911966 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.912485 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.912637 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:21.919569 (+ 6932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:39519: BlockingRecv error: recv got EOF from 127.16.21.3:39519 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 10:48:21 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:21 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:21.983489 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:21.975627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:43623)
0806 10:48:21.976284 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:21.976300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:21.976407 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:21.976687 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:21.976690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:21.977113 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:21.977307 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:21.977316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.978254 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.978258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:21.979160 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:21.979171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.979834 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.979837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:21.979964 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:21.980518 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:21.980536 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:21.983250 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":282,"threads_started":1}
W20260806 10:48:21.983628 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:21.975685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:43623 (local address 127.16.21.62:35175)
0806 10:48:21.975851 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:21.975855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:21.976365 (+ 510us) server_negotiation.cc:407] Connection header received
0806 10:48:21.976502 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:21.976505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:21.976563 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:21.976639 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:21.977435 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.978125 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.979277 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:21.979706 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:21.979799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:21.983420 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:43623: BlockingRecv error: recv got EOF from 127.16.21.1:43623 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 10:48:22 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:22 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:22.448660 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:22.440327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:60765)
0806 10:48:22.440958 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:22.440974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:22.441083 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:22.441364 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:22.441368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:22.441766 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:22.441931 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:22.441939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.442889 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.442893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:22.443832 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:22.443850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.444661 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.444665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:22.444783 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:22.445335 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:22.445351 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:22.448408 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":269,"threads_started":1}
W20260806 10:48:22.448849 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:22.440408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:60765 (local address 127.16.21.62:35175)
0806 10:48:22.440649 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:22.440654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:22.441046 (+ 392us) server_negotiation.cc:407] Connection header received
0806 10:48:22.441181 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:22.441185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:22.441247 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:22.441328 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:22.442055 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.442761 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.443976 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.444531 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.444630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:22.448620 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:60765: BlockingRecv error: recv got EOF from 127.16.21.2:60765 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 06 10:48:22 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:22 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:48:22.929183 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:22.920682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:47575 (local address 127.16.21.62:35175)
0806 10:48:22.920858 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:22.920862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:22.921343 (+ 481us) server_negotiation.cc:407] Connection header received
0806 10:48:22.921477 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:22.921480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:22.921552 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:22.921632 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:22.922407 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.923114 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.924480 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.924905 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.925102 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:22.928900 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:47575: BlockingRecv error: recv got EOF from 127.16.21.3:47575 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:48:22.929222 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:22.920620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:47575)
0806 10:48:22.921253 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:22.921275 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:22.921386 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:22.921670 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:22.921674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:22.922075 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:22.922288 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:22.922298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.923238 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.923242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:22.924359 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:22.924372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.925048 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.925053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:22.925211 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:22.925873 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:22.925890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:22.928704 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":282,"threads_started":1}
Aug 06 10:48:22 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:22 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:22.992549 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:22.984430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:36647)
0806 10:48:22.984931 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:22.984947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:22.985054 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:22.985346 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:22.985349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:22.985732 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:22.985898 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:22.985905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.986903 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.986907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:22.987889 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:22.987903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.988889 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.988895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:22.989045 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:22.989661 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:22.989678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:22.992317 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
W20260806 10:48:22.992730 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:22.984592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:36647 (local address 127.16.21.62:35175)
0806 10:48:22.984801 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:22.984807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:22.985015 (+ 208us) server_negotiation.cc:407] Connection header received
0806 10:48:22.985152 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:22.985156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:22.985216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:22.985297 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:22.986009 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.986788 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.988045 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:22.988701 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:22.988805 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:22.992500 (+ 3695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:36647: BlockingRecv error: recv got EOF from 127.16.21.1:36647 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 10:48:23 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:23 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:23.458896 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:23.449607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:35203)
0806 10:48:23.450174 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:23.450191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:23.450299 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:23.450634 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:23.450637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:23.451265 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:23.451423 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:23.451433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.452932 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.452936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:23.453858 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:23.453870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.454589 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.454593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:23.454717 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:23.455407 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:23.455436 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:23.458640 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":306,"threads_started":1}
W20260806 10:48:23.459160 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:23.449845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:35203 (local address 127.16.21.62:35175)
0806 10:48:23.450008 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:23.450012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:23.450260 (+ 248us) server_negotiation.cc:407] Connection header received
0806 10:48:23.450391 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:23.450395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:23.450510 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:23.450596 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:23.451643 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.452798 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.453983 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.454436 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.454533 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:23.458896 (+ 4363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:35203: BlockingRecv error: recv got EOF from 127.16.21.2:35203 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 10:48:23 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:23 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:23.938709 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:23.930419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:56175)
0806 10:48:23.930879 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:23.930895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:23.931014 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:23.931299 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:23.931302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:23.931685 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:23.931846 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:23.931855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.932956 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.932960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:23.934194 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:23.934207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.934875 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.934879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:23.935048 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:23.935693 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:23.935711 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:23.938491 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
W20260806 10:48:23.938923 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:23.930473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:56175 (local address 127.16.21.62:35175)
0806 10:48:23.930688 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:23.930695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:23.930980 (+ 285us) server_negotiation.cc:407] Connection header received
0806 10:48:23.931106 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:23.931110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:23.931171 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:23.931249 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:23.931964 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.932831 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.934316 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.934718 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.934818 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:23.938671 (+ 3853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:56175: BlockingRecv error: recv got EOF from 127.16.21.3:56175 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 10:48:23 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:23 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:24.001684 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:23.993491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:45385)
0806 10:48:23.993983 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:23.993998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:23.994145 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:23.994493 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:23.994500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:23.994905 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:23.995081 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:23.995090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.996054 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.996064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:23.997061 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:23.997074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.997760 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.997766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:23.997949 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:23.998467 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:23.998485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:24.001324 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
W20260806 10:48:24.001760 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:23.993521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:45385 (local address 127.16.21.62:35175)
0806 10:48:23.993722 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:23.993729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:23.994077 (+ 348us) server_negotiation.cc:407] Connection header received
0806 10:48:23.994268 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:23.994285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:23.994355 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:23.994452 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:23.995217 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.995921 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.997193 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:23.997612 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:23.997739 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:24.001529 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:45385: BlockingRecv error: recv got EOF from 127.16.21.1:45385 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 10:48:24 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:24 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:24.467897 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:24.459802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:45859)
0806 10:48:24.460328 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:24.460348 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:24.460463 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:24.460746 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:24.460749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:24.461135 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:24.461311 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:24.461320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.462302 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.462305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:24.463270 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:24.463281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.463988 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.463992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:24.464116 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:24.464742 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:24.464761 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:24.467544 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
W20260806 10:48:24.467986 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:24.459864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45859 (local address 127.16.21.62:35175)
0806 10:48:24.460088 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:24.460094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:24.460430 (+ 336us) server_negotiation.cc:407] Connection header received
0806 10:48:24.460565 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:24.460569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:24.460632 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:24.460711 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:24.461428 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.462182 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.463386 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.463857 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.463953 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:24.467730 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:45859: BlockingRecv error: recv got EOF from 127.16.21.2:45859 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 10:48:24 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:24 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:24.947909 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:24.939644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:55659)
0806 10:48:24.940164 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:24.940179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:24.940398 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:24.940738 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:24.940743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:24.941156 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:24.941337 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:24.941347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.942287 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.942293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:24.943157 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:24.943167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.943936 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.943942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:24.944116 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:24.944766 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:24.944790 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:24.947665 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":282,"threads_started":1}
W20260806 10:48:24.948071 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:24.939693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:55659 (local address 127.16.21.62:35175)
0806 10:48:24.939947 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:24.939955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:24.940285 (+ 330us) server_negotiation.cc:407] Connection header received
0806 10:48:24.940534 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:24.940538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:24.940607 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:24.940706 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:24.941490 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.942147 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.943300 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:24.943737 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:24.943897 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:24.947847 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:55659: BlockingRecv error: recv got EOF from 127.16.21.3:55659 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 10:48:25 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:25 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:25.011039 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:25.002925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:41743)
0806 10:48:25.003458 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:25.003473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:25.003617 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:25.004102 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:25.004106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:25.004492 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:25.004644 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:25.004652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.005656 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.005661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:25.006610 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:25.006621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.007334 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.007340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:25.007525 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:25.008056 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:25.008074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:25.010709 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
W20260806 10:48:25.011179 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:25.002977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:41743 (local address 127.16.21.62:35175)
0806 10:48:25.003191 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:25.003200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:25.003555 (+ 355us) server_negotiation.cc:407] Connection header received
0806 10:48:25.003739 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:25.003744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:25.003817 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:25.003917 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:25.004791 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.005510 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.006749 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.007181 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.007305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:25.010925 (+ 3620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:41743: BlockingRecv error: recv got EOF from 127.16.21.1:41743 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 10:48:25 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:25 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:25.477486 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:25.468929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:43247)
0806 10:48:25.469470 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:25.469486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:25.469631 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:25.469962 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:25.469967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:25.470373 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:25.470515 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:25.470523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.471530 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.471536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:25.472593 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:25.472605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.473326 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.473332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:25.473501 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:25.474262 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:25.474286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:25.477214 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":272,"threads_started":1}
W20260806 10:48:25.477707 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:25.468977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:43247 (local address 127.16.21.62:35175)
0806 10:48:25.469177 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:25.469183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:25.469564 (+ 381us) server_negotiation.cc:407] Connection header received
0806 10:48:25.469757 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:25.469761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:25.469829 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:25.469929 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:25.470662 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.471386 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.472741 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.473157 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.473283 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:25.477435 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:43247: BlockingRecv error: recv got EOF from 127.16.21.2:43247 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 10:48:25 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:25 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:25.957706 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:25.948898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:58355)
0806 10:48:25.949430 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:25.949452 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:25.949587 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:25.950170 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:25.950174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:25.950757 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:25.950975 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:25.950985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.952091 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.952097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:25.953087 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:25.953099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.953821 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.953827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:25.954028 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:25.954567 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:25.954583 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:25.957365 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
W20260806 10:48:25.957844 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:25.949208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:58355 (local address 127.16.21.62:35175)
0806 10:48:25.949490 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:25.949498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:25.949519 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:48:25.949741 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:25.949746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:25.949852 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:25.949931 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:25.951135 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.951955 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.953229 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:25.953657 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:25.953804 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:25.957581 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:58355: BlockingRecv error: recv got EOF from 127.16.21.3:58355 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Aug 06 10:48:26 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:26 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:26.020911 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:26.012017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:49577)
0806 10:48:26.012601 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:26.012618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:26.012730 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:26.012998 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:26.013001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:26.013427 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:26.013609 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:26.013618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.014970 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.014977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:26.016061 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:26.016074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.016804 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.016808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:26.016924 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:26.017553 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:26.017571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:26.020546 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":327,"threads_started":1}
W20260806 10:48:26.020963 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:26.012081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:49577 (local address 127.16.21.62:35175)
0806 10:48:26.012250 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:26.012294 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:26.012688 (+ 394us) server_negotiation.cc:407] Connection header received
0806 10:48:26.012840 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:26.012843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:26.012903 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:26.012983 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:26.013733 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.014820 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.016192 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.016661 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.016756 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:26.020744 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:49577: BlockingRecv error: recv got EOF from 127.16.21.1:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 10:48:26 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:26 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:26.487011 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:26.478330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:50757)
0806 10:48:26.478900 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:26.478922 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:26.479095 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:26.479533 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:26.479540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:26.479971 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:26.480144 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:26.480152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.481271 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.481275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:26.482381 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:26.482394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.483145 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.483150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:26.483295 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:26.483870 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:26.483888 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:26.486750 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":267,"threads_started":1}
W20260806 10:48:26.487251 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:26.478372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:50757 (local address 127.16.21.62:35175)
0806 10:48:26.478587 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:26.478594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:26.479033 (+ 439us) server_negotiation.cc:407] Connection header received
0806 10:48:26.479234 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:26.479239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:26.479346 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:26.479450 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:26.480307 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.481108 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.482540 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.483009 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.483098 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:26.486946 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:50757: BlockingRecv error: recv got EOF from 127.16.21.2:50757 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 10:48:26 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:26 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:26.971688 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:26.958688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:37881)
0806 10:48:26.959263 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:26.959279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:26.959429 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:26.959757 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:26.959761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:26.960188 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:26.960446 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:26.960457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.961490 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.961495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:26.962392 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:26.962404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.963123 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.963130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:26.963302 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:26.968494 (+ 5192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:26.968512 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:26.971329 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":347,"threads_started":1}
W20260806 10:48:26.971768 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:26.958758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:37881 (local address 127.16.21.62:35175)
0806 10:48:26.958996 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:26.959005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:26.959359 (+ 354us) server_negotiation.cc:407] Connection header received
0806 10:48:26.959543 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:26.959548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:26.959623 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:26.959726 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:26.960612 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.961359 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.962528 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:26.962967 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:26.963092 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:26.971518 (+ 8426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:37881: BlockingRecv error: recv got EOF from 127.16.21.3:37881 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 10:48:27 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:27 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:27.030820 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:27.022084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:46707)
0806 10:48:27.022635 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:27.022651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:27.022758 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:27.023041 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:27.023044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:27.023430 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:27.023655 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:27.023666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.025130 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.025135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:27.026259 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:27.026273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.026976 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.026980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:27.027103 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:27.027730 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:27.027748 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:27.030580 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":307,"threads_started":1}
W20260806 10:48:27.030994 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:27.022247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:46707 (local address 127.16.21.62:35175)
0806 10:48:27.022405 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:27.022409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:27.022716 (+ 307us) server_negotiation.cc:407] Connection header received
0806 10:48:27.022856 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:27.022859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:27.022916 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:27.022995 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:27.023799 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.024984 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.026396 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.026855 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.026947 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:27.030766 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:46707: BlockingRecv error: recv got EOF from 127.16.21.1:46707 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 10:48:27 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:27 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:27.495978 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:27.487862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:45761)
0806 10:48:27.488409 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:27.488424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:27.488537 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:27.488824 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:27.488827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:27.489238 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:27.489404 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:27.489412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.490335 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.490338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:27.491240 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:27.491251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.492141 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.492159 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:27.492402 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:27.493093 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:27.493117 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:27.495752 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":285,"threads_started":1}
W20260806 10:48:27.496127 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:27.487960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45761 (local address 127.16.21.62:35175)
0806 10:48:27.488146 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:27.488150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:27.488496 (+ 346us) server_negotiation.cc:407] Connection header received
0806 10:48:27.488628 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:27.488632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:27.488690 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:27.488771 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:27.489523 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.490223 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.491368 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.492001 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.492120 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:27.495927 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:45761: BlockingRecv error: recv got EOF from 127.16.21.2:45761 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 10:48:27 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:27 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:27.980707 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:27.972678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:44071)
0806 10:48:27.973239 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:27.973254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:27.973402 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:27.973696 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:27.973699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:27.974141 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:27.974310 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:27.974320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.975283 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.975286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:27.976195 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:27.976206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.976989 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.976994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:27.977124 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:27.977669 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:27.977685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:27.980468 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":341,"threads_started":1}
W20260806 10:48:27.980906 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:27.972719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:44071 (local address 127.16.21.62:35175)
0806 10:48:27.972960 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:27.972969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:27.973346 (+ 377us) server_negotiation.cc:407] Connection header received
0806 10:48:27.973514 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:27.973518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:27.973579 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:27.973652 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:27.974433 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.975166 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.976351 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:27.976820 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:27.976907 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:27.980655 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:44071: BlockingRecv error: recv got EOF from 127.16.21.3:44071 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 10:48:28 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:28 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:28.039748 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:28.031742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:57553)
0806 10:48:28.032319 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:28.032335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:28.032444 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:28.032727 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:28.032730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:28.033106 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:28.033271 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:28.033294 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.034226 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.034229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:28.035177 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:28.035188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.035840 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.035844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:28.035955 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:28.036526 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:28.036544 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:28.039392 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":270,"threads_started":1}
W20260806 10:48:28.039857 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:28.031814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:57553 (local address 127.16.21.62:35175)
0806 10:48:28.031968 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:28.031973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:28.032400 (+ 427us) server_negotiation.cc:407] Connection header received
0806 10:48:28.032541 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:28.032545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:28.032604 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:28.032682 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:28.033400 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.034112 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.035297 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.035694 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.035792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:28.039618 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:57553: BlockingRecv error: recv got EOF from 127.16.21.1:57553 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 10:48:28 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:28 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:28.505268 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:28.497006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:47407)
0806 10:48:28.497584 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:28.497599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:28.497743 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:28.498071 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:28.498075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:28.498495 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:28.498652 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:28.498661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.499718 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.499724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:28.500731 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:28.500744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.501509 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.501515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:28.501677 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:28.502237 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:28.502254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:28.505032 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":278,"threads_started":1}
W20260806 10:48:28.506383 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:28.497202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:47407 (local address 127.16.21.62:35175)
0806 10:48:28.497425 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:28.497431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:28.497676 (+ 245us) server_negotiation.cc:407] Connection header received
0806 10:48:28.497862 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:28.497868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:28.497940 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:28.498036 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:28.498800 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.499580 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.500877 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.501331 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.501459 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:28.505260 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:47407: BlockingRecv error: recv got EOF from 127.16.21.2:47407 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 10:48:28 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:28 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:28.989959 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:28.981661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:33877)
0806 10:48:28.982165 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:28.982180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:28.982338 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:28.982740 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:28.982744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:28.983145 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:28.983317 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:28.983326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.984390 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.984396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:28.985355 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:28.985367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.986126 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.986133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:28.986311 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:28.986925 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:28.986943 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:28.989629 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":250,"threads_started":1}
W20260806 10:48:28.990053 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:28.981721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:33877 (local address 127.16.21.62:35175)
0806 10:48:28.981929 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:28.981937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:28.982262 (+ 325us) server_negotiation.cc:407] Connection header received
0806 10:48:28.982482 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:28.982485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:28.982584 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:28.982703 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:28.983460 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.984204 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.985505 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:28.985942 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:28.986075 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:28.989827 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:33877: BlockingRecv error: recv got EOF from 127.16.21.3:33877 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 10:48:29 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:29 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:29.049119 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:29.040873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:56381)
0806 10:48:29.041370 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:29.041385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:29.041538 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:29.041907 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:29.041913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:29.042315 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:29.042494 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:29.042503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.043476 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.043481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:29.044495 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:29.044507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.045215 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.045221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:29.045392 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:29.045992 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:29.046009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:29.048751 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
W20260806 10:48:29.049252 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:29.040965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:56381 (local address 127.16.21.62:35175)
0806 10:48:29.041275 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:29.041281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:29.041468 (+ 187us) server_negotiation.cc:407] Connection header received
0806 10:48:29.041666 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:29.041671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:29.041740 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:29.041867 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:29.042634 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.043330 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.044632 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.045062 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.045199 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:29.048976 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:56381: BlockingRecv error: recv got EOF from 127.16.21.1:56381 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Aug 06 10:48:29 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:29 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:29.514775 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:29.506589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:58947)
0806 10:48:29.507076 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:29.507091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:29.507254 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:29.507713 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:29.507717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:29.508195 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:29.508457 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:29.508467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.509565 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.509569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:29.510492 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:29.510509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.511125 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.511129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:29.511247 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:29.511819 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:29.511835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:29.514521 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":258,"threads_started":1}
W20260806 10:48:29.514916 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:29.506682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58947 (local address 127.16.21.62:35175)
0806 10:48:29.506920 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:29.506927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:29.507180 (+ 253us) server_negotiation.cc:407] Connection header received
0806 10:48:29.507437 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:29.507445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:29.507527 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:29.507647 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:29.508611 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.509435 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.510624 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.511008 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.511098 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:29.514705 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:58947: BlockingRecv error: recv got EOF from 127.16.21.2:58947 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 10:48:29 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:29 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.3@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:29.999006 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:29.990905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.3:34333)
0806 10:48:29.991404 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:29.991420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:29.991532 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:29.991813 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:29.991816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:29.992183 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:29.992449 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:29.992460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.993503 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.993507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:29.994410 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:29.994420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.995073 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.995076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:29.995199 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:29.995775 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:29.995791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:29.998644 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
W20260806 10:48:29.999195 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:29.990950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:34333 (local address 127.16.21.62:35175)
0806 10:48:29.991142 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:29.991147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:29.991493 (+ 346us) server_negotiation.cc:407] Connection header received
0806 10:48:29.991629 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:29.991632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:29.991702 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:29.991798 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:29.992588 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.993373 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.994526 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:29.994954 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:29.995053 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:29.998879 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.3:34333: BlockingRecv error: recv got EOF from 127.16.21.3:34333 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 10:48:30 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:30 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.1@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:30.059276 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:30.050307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.1:46245)
0806 10:48:30.050839 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:30.050855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:30.051019 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:30.051388 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:30.051394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:30.051828 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:30.052020 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:30.052030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.053122 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.053128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:30.054055 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:30.054067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.054810 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.054815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:30.055004 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.055574 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:30.055592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:30.058871 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":289,"threads_started":1}
W20260806 10:48:30.059432 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:30.050381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:46245 (local address 127.16.21.62:35175)
0806 10:48:30.050653 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:30.050662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:30.050937 (+ 275us) server_negotiation.cc:407] Connection header received
0806 10:48:30.051175 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:30.051187 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:30.051249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:30.051349 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:30.052187 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.052978 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.054193 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.054661 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.054804 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.059144 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.1:46245: BlockingRecv error: recv got EOF from 127.16.21.1:46245 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 10:48:30 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:48:30 dist-test-slave-0b58 krb5kdc[24861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.21.2@KRBTEST.COM for oryx/127.16.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:48:30.524292 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:30.515838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:35175 (local address 127.16.21.2:37549)
0806 10:48:30.516381 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:30.516405 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:30.516532 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:30.516945 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:30.516950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:30.517530 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:30.517854 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:30.517870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.518934 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.518938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:30.519839 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:30.519850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.520507 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.520511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:30.520669 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.521243 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:30.521260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:30.523920 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.21.62:35175: Server oryx/127.16.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
W20260806 10:48:30.524406 25739 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:30.516035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:37549 (local address 127.16.21.62:35175)
0806 10:48:30.516288 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:30.516296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:30.516504 (+ 208us) server_negotiation.cc:407] Connection header received
0806 10:48:30.516687 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:30.516691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:30.516751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:30.516880 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:30.518037 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.518819 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.519951 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.520375 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.520464 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.524111 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.21.2:37549: BlockingRecv error: recv got EOF from 127.16.21.2:37549 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:48:30.822202 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25453
I20260806 10:48:30.834285 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:30.823218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.16.21.1:46717)
0806 10:48:30.823750 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:30.823758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:30.823785 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:48:30.823843 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:30.823849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:30.824096 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:30.824244 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:30.825864 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.826885 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.828091 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.828694 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.828790 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.831305 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:30.831314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:30.831326 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:30.831358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:30.833802 (+ 2444us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 10:48:30.834044 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34102: Request ticket server kudu/127.16.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
W20260806 10:48:30.834405 25918 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34102: Request ticket server kudu/127.16.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 10:48:30.834427 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25453: Not authorized: Client connection negotiation failed: client connection to 127.16.21.1:46717: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:48:30.834627 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25453
I20260806 10:48:30.856732 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25605
I20260806 10:48:30.868709 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:30.857638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.16.21.2:45827)
0806 10:48:30.858229 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:30.858236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:30.858270 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:48:30.858335 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:30.858341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:30.858550 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:30.858716 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:30.859902 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.861045 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.862274 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.862813 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.862906 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.865687 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:30.865695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:30.865708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:30.865741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:30.868179 (+ 2438us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 10:48:30.868480 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55530: Request ticket server kudu/127.16.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
W20260806 10:48:30.868844 25919 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55530: Request ticket server kudu/127.16.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 10:48:30.868837 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25605: Not authorized: Client connection negotiation failed: client connection to 127.16.21.2:45827: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:48:30.868975 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25605
I20260806 10:48:30.888962 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25738
W20260806 10:48:30.900694 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25738: Not authorized: Client connection negotiation failed: client connection to 127.16.21.3:41417: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:48:30.900739 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:30.889913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.16.21.3:41417)
0806 10:48:30.890508 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:30.890515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:30.890541 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:48:30.890606 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:30.890612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:30.890855 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:30.891003 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:30.892213 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.893299 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.894473 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:30.895020 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:30.895113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:30.897513 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:30.897522 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:30.897534 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:30.897569 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:30.900048 (+ 2479us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 10:48:30.900422 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33896: Request ticket server kudu/127.16.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":334,"threads_started":1}
W20260806 10:48:30.900856 25920 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33896: Request ticket server kudu/127.16.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:48:30.900887 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25738
I20260806 10:48:30.920797 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25383
I20260806 10:48:30.921998 25446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:31.043495 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25383
2026-08-06T10:48:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21161 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T10:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:31Z Disabled control of system clock
I20260806 10:48:31.138085 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36523
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 10:48:31.470957 25925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:31.471230 25925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:31.471310 25925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:31.478413 25925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:31.478513 25925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:31.478550 25925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:31.478610 25925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:31.478655 25925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:31.487468 25925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:36523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:31.488668 25925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:31.490377 25925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:31.502375 25930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:31.502296 25933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:31.502346 25931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:31.503136 25925 server_base.cc:1034] running on GCE node
I20260806 10:48:31.503832 25925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:31.505144 25925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:31.506376 25925 hybrid_clock.cc:648] HybridClock initialized: now 1786013311506344 us; error 43 us; skew 500 ppm
I20260806 10:48:31.506735 25925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:31.509052 25925 webserver.cc:460] Webserver started at http://127.16.21.62:36265/ using document root <none> and password file <none>
I20260806 10:48:31.509716 25925 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:31.509850 25925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:31.510174 25925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:31.513046 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "47d7b98a25f0442ebe342c2a21052de6"
format_stamp: "Formatted at 2026-08-06 10:48:31 on dist-test-slave-0b58"
server_key: "c91eaad9c7b5981a6027e2e7024896eb"
server_key_iv: "013739563c8bd870f168262c742514fc"
server_key_version: "encryptionkey@0"
I20260806 10:48:31.513813 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "47d7b98a25f0442ebe342c2a21052de6"
format_stamp: "Formatted at 2026-08-06 10:48:31 on dist-test-slave-0b58"
server_key: "c91eaad9c7b5981a6027e2e7024896eb"
server_key_iv: "013739563c8bd870f168262c742514fc"
server_key_version: "encryptionkey@0"
I20260806 10:48:31.518702 25925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:48:31.522080 25939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:31.523365 25925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:48:31.523558 25925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "47d7b98a25f0442ebe342c2a21052de6"
format_stamp: "Formatted at 2026-08-06 10:48:31 on dist-test-slave-0b58"
server_key: "c91eaad9c7b5981a6027e2e7024896eb"
server_key_iv: "013739563c8bd870f168262c742514fc"
server_key_version: "encryptionkey@0"
I20260806 10:48:31.523784 25925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:31.557071 25925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:31.557931 25925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:31.558243 25925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:31.577719 25925 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:36523
I20260806 10:48:31.577766 25991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:36523 every 8 connection(s)
I20260806 10:48:31.579036 25925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:48:31.581014 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 25925
I20260806 10:48:31.581207 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:48:31.582094 16468 external_mini_cluster.cc:1428] Setting key e33480f3ed9fb2304a0dc8cd2862bcc1
I20260806 10:48:31.583535 25992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:31.595707 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:31.584133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.16.21.62:36523)
0806 10:48:31.585152 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:31.585163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:31.585210 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:48:31.586740 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:31.586759 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:31.587198 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:31.587514 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:31.588187 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:31.589511 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:31.591407 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:31.592015 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:31.592150 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:31.593466 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:31.593476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:31.593487 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:31.594030 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:31.594106 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:31.594131 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:31.594232 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:31.594473 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 10:48:31.594823 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":395,"threads_started":1}
I20260806 10:48:31.597198 25992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6: Bootstrap starting.
I20260806 10:48:31.600198 25992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:31.601177 25992 log.cc:824] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:31.603927 25992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6: No bootstrap required, opened a new log
I20260806 10:48:31.611050 25992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7b98a25f0442ebe342c2a21052de6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36523 } }
I20260806 10:48:31.611413 25992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:31.611524 25992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d7b98a25f0442ebe342c2a21052de6, State: Initialized, Role: FOLLOWER
I20260806 10:48:31.612128 25992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7b98a25f0442ebe342c2a21052de6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36523 } }
I20260806 10:48:31.612426 25992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:48:31.612571 25992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:48:31.612754 25992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:31.615013 25992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7b98a25f0442ebe342c2a21052de6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36523 } }
I20260806 10:48:31.615482 25992 leader_election.cc:302] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d7b98a25f0442ebe342c2a21052de6; no voters:
I20260806 10:48:31.616016 25992 leader_election.cc:288] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:48:31.616140 25997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:31.617012 25997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [term 1 LEADER]: Becoming Leader. State: Replica: 47d7b98a25f0442ebe342c2a21052de6, State: Running, Role: LEADER
I20260806 10:48:31.617547 25997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7b98a25f0442ebe342c2a21052de6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36523 } }
I20260806 10:48:31.618062 25992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:48:31.620429 25998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d7b98a25f0442ebe342c2a21052de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7b98a25f0442ebe342c2a21052de6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36523 } } }
I20260806 10:48:31.620429 25999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d7b98a25f0442ebe342c2a21052de6. Latest consensus state: current_term: 1 leader_uuid: "47d7b98a25f0442ebe342c2a21052de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7b98a25f0442ebe342c2a21052de6" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36523 } } }
I20260806 10:48:31.620951 25998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:31.620949 25999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:31.623106 26003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:31.629212 26003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:31.638916 26003 catalog_manager.cc:1347] Generated new cluster ID: 3d81b1f3df6e47c9afe232eacf6dd330
I20260806 10:48:31.639021 26003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:31.651041 26003 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:48:31.652884 26003 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:31.667997 26003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d7b98a25f0442ebe342c2a21052de6: Generated new TSK 0
I20260806 10:48:31.668855 26003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:48:31.681766 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36523
--builtin_ntp_servers=127.16.21.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 10:48:32.031517 26016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:32.031777 26016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:32.031937 26016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:32.039769 26016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:32.039857 26016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:32.040076 26016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:48:32.049129 26016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:32.050313 26016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:32.052043 26016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:32.064877 26021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:32.064817 26024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:32.064910 26022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:32.065682 26016 server_base.cc:1034] running on GCE node
I20260806 10:48:32.066411 26016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:32.067552 26016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:32.068910 26016 hybrid_clock.cc:648] HybridClock initialized: now 1786013312068856 us; error 33 us; skew 500 ppm
I20260806 10:48:32.069274 26016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:32.076393 26016 webserver.cc:460] Webserver started at http://127.16.21.1:37971/ using document root <none> and password file <none>
I20260806 10:48:32.077064 26016 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:32.077226 26016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:32.077574 26016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:32.080343 26016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "02f3c64ee10c4413a7d22516e37d50e8"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "3cce2a89a0376c1391b1936066d7cd6e"
server_key_iv: "03a45378c07dbcc7849b59c815e046a5"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.081183 26016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "02f3c64ee10c4413a7d22516e37d50e8"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "3cce2a89a0376c1391b1936066d7cd6e"
server_key_iv: "03a45378c07dbcc7849b59c815e046a5"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.086079 26016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:48:32.089435 26030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:32.090624 26016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260806 10:48:32.090794 26016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "02f3c64ee10c4413a7d22516e37d50e8"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "3cce2a89a0376c1391b1936066d7cd6e"
server_key_iv: "03a45378c07dbcc7849b59c815e046a5"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.091022 26016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:32.108196 26016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:32.109002 26016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:32.109261 26016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:32.110102 26016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:32.111431 26016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:32.111562 26016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:32.111687 26016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:32.111778 26016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:32.145792 26016 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:43395
I20260806 10:48:32.145872 26142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:43395 every 8 connection(s)
I20260806 10:48:32.147048 26016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:48:32.155893 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26016
I20260806 10:48:32.156085 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:48:32.156831 16468 external_mini_cluster.cc:1428] Setting key 16e400a38a1d4639bb9bb94a4cfde744
I20260806 10:48:32.159339 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:32.149505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:36353 (local address 127.16.21.62:36523)
0806 10:48:32.149971 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:32.149976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:32.150966 (+ 990us) server_negotiation.cc:407] Connection header received
0806 10:48:32.152813 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:32.152820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:32.152904 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:32.153025 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:32.154063 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.154822 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.156523 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.156963 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.157102 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:32.158472 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:32.158478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:32.158483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:32.158509 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:32.158548 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:32.158561 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:32.158658 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:32.158894 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 10:48:32.158985 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":262,"threads_started":1}
I20260806 10:48:32.159420 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36523
--builtin_ntp_servers=127.16.21.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 10:48:32.160223 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:32.149948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36523 (local address 127.16.21.1:36353)
0806 10:48:32.150782 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:32.150823 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:32.152527 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:32.153250 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:32.153258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:32.153335 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:32.153885 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:32.153901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.154979 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.154990 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:32.156382 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:32.156394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.157198 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.157207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:32.157443 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:32.158164 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:48:32.158168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:32.158205 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:48:32.158275 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:32.158657 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:32.158679 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:32.158689 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:48:32.158881 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 10:48:32.159236 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":269,"threads_started":1}
I20260806 10:48:32.161885 26143 heartbeater.cc:347] Connected to a master server at 127.16.21.62:36523
I20260806 10:48:32.162189 26143 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:32.162914 26143 heartbeater.cc:511] Master 127.16.21.62:36523 requested a full tablet report, sending...
I20260806 10:48:32.165146 25956 ts_manager.cc:194] Registered new tserver with Master: 02f3c64ee10c4413a7d22516e37d50e8 (127.16.21.1:43395)
I20260806 10:48:32.167407 25956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.1:36353
W20260806 10:48:32.496080 26147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:32.496419 26147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:32.496624 26147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:32.503715 26147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:32.503800 26147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:32.503904 26147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:48:32.512753 26147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:32.513876 26147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:32.515594 26147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:32.529348 26155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:32.529390 26153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:32.529474 26152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:32.530462 26147 server_base.cc:1034] running on GCE node
I20260806 10:48:32.531075 26147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:32.531961 26147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:32.533187 26147 hybrid_clock.cc:648] HybridClock initialized: now 1786013312533147 us; error 52 us; skew 500 ppm
I20260806 10:48:32.533519 26147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:32.535789 26147 webserver.cc:460] Webserver started at http://127.16.21.2:42117/ using document root <none> and password file <none>
I20260806 10:48:32.536532 26147 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:32.536664 26147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:32.536989 26147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:32.539738 26147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "4840731169a441c09543ff4fbfbe9533"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "19dbd714a1da98d3d9c59069ca0fe3a3"
server_key_iv: "36aea0d647769bfeafa8f56d8aa4069a"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.540616 26147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "4840731169a441c09543ff4fbfbe9533"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "19dbd714a1da98d3d9c59069ca0fe3a3"
server_key_iv: "36aea0d647769bfeafa8f56d8aa4069a"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.545639 26147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 10:48:32.549036 26161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:32.550307 26147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:48:32.550485 26147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "4840731169a441c09543ff4fbfbe9533"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "19dbd714a1da98d3d9c59069ca0fe3a3"
server_key_iv: "36aea0d647769bfeafa8f56d8aa4069a"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.550712 26147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:32.569742 26147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:32.570571 26147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:32.570863 26147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:32.571858 26147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:32.573315 26147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:32.573444 26147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:32.573575 26147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:32.573673 26147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:32.611536 26147 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:34533
I20260806 10:48:32.611622 26273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:34533 every 8 connection(s)
I20260806 10:48:32.612895 26147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:48:32.618644 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26147
I20260806 10:48:32.618830 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:48:32.619366 16468 external_mini_cluster.cc:1428] Setting key 33f1fd3e8bf0b2f9f3efba43e025c989
I20260806 10:48:32.622243 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36523
--builtin_ntp_servers=127.16.21.20:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 10:48:32.625530 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:32.615374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:37949 (local address 127.16.21.62:36523)
0806 10:48:32.615593 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:32.615599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:32.616933 (+ 1334us) server_negotiation.cc:407] Connection header received
0806 10:48:32.618711 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:32.618720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:32.618837 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:32.618967 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:32.620079 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.621184 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.622810 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.623195 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.623328 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:32.624706 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:32.624711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:32.624714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:32.624731 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:32.624768 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:32.624780 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:32.624874 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:32.625088 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 10:48:32.625183 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:48:32.626494 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:32.615896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36523 (local address 127.16.21.2:37949)
0806 10:48:32.616797 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:32.616832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:32.618456 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:32.619208 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:32.619215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:32.619300 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:32.619905 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:32.619922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.621354 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.621358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:32.622658 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:32.622675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:32.623414 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:32.623420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:32.623643 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:32.624402 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:48:32.624404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:32.624440 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:48:32.624508 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:32.624874 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:32.624888 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:32.624898 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:48:32.625075 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 10:48:32.625454 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":306,"threads_started":1}
I20260806 10:48:32.627914 26274 heartbeater.cc:347] Connected to a master server at 127.16.21.62:36523
I20260806 10:48:32.628244 26274 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:32.629182 26274 heartbeater.cc:511] Master 127.16.21.62:36523 requested a full tablet report, sending...
I20260806 10:48:32.630653 25956 ts_manager.cc:194] Registered new tserver with Master: 4840731169a441c09543ff4fbfbe9533 (127.16.21.2:34533)
I20260806 10:48:32.631879 25956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.2:37949
W20260806 10:48:32.952953 26277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:32.953186 26277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:32.953311 26277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:32.960399 26277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:32.960482 26277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:32.960623 26277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:48:32.969462 26277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:32.970623 26277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:32.972348 26277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:32.988848 26283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:32.989090 26282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:32.988869 26285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:32.989478 26277 server_base.cc:1034] running on GCE node
I20260806 10:48:32.990144 26277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:32.991084 26277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:32.992309 26277 hybrid_clock.cc:648] HybridClock initialized: now 1786013312992220 us; error 53 us; skew 500 ppm
I20260806 10:48:32.992609 26277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:32.994906 26277 webserver.cc:460] Webserver started at http://127.16.21.3:39825/ using document root <none> and password file <none>
I20260806 10:48:32.995594 26277 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:32.995733 26277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:32.996080 26277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:32.998857 26277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "cd78a60ff1c148eeaa235d49032ba225"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "56029347221222f49fc153c2376c3d9a"
server_key_iv: "6952536dcec482cc5c037c12e95bfc57"
server_key_version: "encryptionkey@0"
I20260806 10:48:32.999615 26277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "cd78a60ff1c148eeaa235d49032ba225"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "56029347221222f49fc153c2376c3d9a"
server_key_iv: "6952536dcec482cc5c037c12e95bfc57"
server_key_version: "encryptionkey@0"
I20260806 10:48:33.004724 26277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:48:33.008189 26291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:33.009583 26277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:48:33.009753 26277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "cd78a60ff1c148eeaa235d49032ba225"
format_stamp: "Formatted at 2026-08-06 10:48:32 on dist-test-slave-0b58"
server_key: "56029347221222f49fc153c2376c3d9a"
server_key_iv: "6952536dcec482cc5c037c12e95bfc57"
server_key_version: "encryptionkey@0"
I20260806 10:48:33.009958 26277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:33.035624 26277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:33.036499 26277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:33.036782 26277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:33.037729 26277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:33.039079 26277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:33.039197 26277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:33.039341 26277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:33.039433 26277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:33.072678 26277 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:35165
I20260806 10:48:33.072757 26403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:35165 every 8 connection(s)
I20260806 10:48:33.073921 26277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:48:33.081821 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26277
I20260806 10:48:33.081993 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:48:33.082495 16468 external_mini_cluster.cc:1428] Setting key 7c28b96d083808deb5eb79e81d4617b0
I20260806 10:48:33.085304 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:33.076128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:59925 (local address 127.16.21.62:36523)
0806 10:48:33.076375 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:33.076381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:33.077653 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 10:48:33.079214 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:33.079219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:33.079285 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:33.079363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:33.080299 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.081020 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.082624 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.083033 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.083174 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:33.084484 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:33.084489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:33.084492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:33.084508 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:33.084539 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:33.084550 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:33.084653 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:33.084866 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 10:48:33.084956 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:48:33.086222 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:33.076655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36523 (local address 127.16.21.3:59925)
0806 10:48:33.077508 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:33.077541 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:33.079005 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:33.079535 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:33.079542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:33.079619 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:33.080126 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:33.080141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.081148 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.081152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:33.082479 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:33.082491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.083260 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.083267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:33.083484 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:33.084144 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:48:33.084147 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:33.084180 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:48:33.084247 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:33.084654 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:33.084667 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:33.084675 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:48:33.084856 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 10:48:33.085183 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":249,"threads_started":1}
I20260806 10:48:33.087370 26404 heartbeater.cc:347] Connected to a master server at 127.16.21.62:36523
I20260806 10:48:33.087690 26404 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:33.088464 26404 heartbeater.cc:511] Master 127.16.21.62:36523 requested a full tablet report, sending...
I20260806 10:48:33.089730 25956 ts_manager.cc:194] Registered new tserver with Master: cd78a60ff1c148eeaa235d49032ba225 (127.16.21.3:35165)
I20260806 10:48:33.090457 25956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.3:59925
I20260806 10:48:33.097316 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 10:48:33.103446 26145 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:33.101922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.16.21.62:36523)
0806 10:48:33.102110 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:33.102119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:33.102577 (+ 458us) server_negotiation.cc:407] Connection header received
0806 10:48:33.102705 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:33.102709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:33.102760 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:33.102838 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:33.103221 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47622: BlockingRecv error: recv got EOF from 127.0.0.1:47622 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:48:33.103849 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26016
I20260806 10:48:33.110846 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:33.104793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.16.21.1:43395)
0806 10:48:33.105258 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:33.105263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:33.105296 (+ 33us) server_negotiation.cc:407] Connection header received
0806 10:48:33.105346 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:33.105351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:33.105518 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:33.105659 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:33.106019 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.106986 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.108154 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.108710 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.108803 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:33.109682 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:33.109689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:33.109699 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:33.110185 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:33.110224 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:33.110241 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:33.110342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:33.110460 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 10:48:33.110567 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260806 10:48:33.111899 26138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:33.192276 26143 heartbeater.cc:503] Master 127.16.21.62:36523 was elected leader, sending a full tablet report...
I20260806 10:48:33.200197 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26016
I20260806 10:48:33.219031 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26147
I20260806 10:48:33.227006 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:33.219903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.16.21.2:34533)
0806 10:48:33.220536 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:33.220542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:33.220566 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:48:33.220620 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:33.220625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:33.220821 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:33.221007 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:33.221409 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.222513 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.223727 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.224304 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.224445 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:33.225510 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:33.225517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:33.225529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:33.226128 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:33.226184 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:33.226219 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:33.226378 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:33.226490 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 10:48:33.226631 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":386,"threads_started":1}
I20260806 10:48:33.228431 26269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:33.309830 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26147
I20260806 10:48:33.329090 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26277
I20260806 10:48:33.336524 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:33.330077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.16.21.3:35165)
0806 10:48:33.330703 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:33.330710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:33.330757 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:48:33.330824 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:33.330830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:33.331032 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:33.331152 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:33.331534 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.332647 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.333777 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:33.334299 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:33.334386 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:33.335284 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:33.335291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:33.335301 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:33.335797 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:33.335839 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:33.335856 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:33.335945 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:33.336024 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 10:48:33.336119 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":335,"threads_started":1}
I20260806 10:48:33.337652 26399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:33.419646 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26277
I20260806 10:48:33.441300 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 25925
I20260806 10:48:33.442580 25986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:33.547060 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 25925
2026-08-06T10:48:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2495 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T10:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:33Z Disabled control of system clock
I20260806 10:48:33.633071 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:42161
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:42161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 10:48:33.955247 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:33.955499 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:33.955579 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:33.962451 26423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:33.962543 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:33.962587 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:33.962635 26423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:33.962677 26423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:33.971482 26423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:42161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:42161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:33.972642 26423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:33.974370 26423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:33.985867 26429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:33.985927 26430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:33.985954 26432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:33.986789 26423 server_base.cc:1034] running on GCE node
I20260806 10:48:33.987510 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:33.988870 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:33.990229 26423 hybrid_clock.cc:648] HybridClock initialized: now 1786013313990197 us; error 44 us; skew 500 ppm
I20260806 10:48:33.990600 26423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:33.992996 26423 webserver.cc:460] Webserver started at http://127.16.21.62:38245/ using document root <none> and password file <none>
I20260806 10:48:33.993675 26423 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:33.993847 26423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:33.994184 26423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:33.997094 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "f75176fcd4054b3a9ff5bb7653184a86"
format_stamp: "Formatted at 2026-08-06 10:48:33 on dist-test-slave-0b58"
server_key: "003d93e82777638cc452e810bff4320e"
server_key_iv: "1d57c9b325def0056bb294fec8e61e5a"
server_key_version: "encryptionkey@0"
I20260806 10:48:33.997927 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "f75176fcd4054b3a9ff5bb7653184a86"
format_stamp: "Formatted at 2026-08-06 10:48:33 on dist-test-slave-0b58"
server_key: "003d93e82777638cc452e810bff4320e"
server_key_iv: "1d57c9b325def0056bb294fec8e61e5a"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.002794 26423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:48:34.006124 26438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:34.007378 26423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:48:34.007560 26423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "f75176fcd4054b3a9ff5bb7653184a86"
format_stamp: "Formatted at 2026-08-06 10:48:33 on dist-test-slave-0b58"
server_key: "003d93e82777638cc452e810bff4320e"
server_key_iv: "1d57c9b325def0056bb294fec8e61e5a"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.007789 26423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:34.013388 26423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:34.014128 26423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:34.014432 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:34.033072 26423 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:42161
I20260806 10:48:34.033116 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:42161 every 8 connection(s)
I20260806 10:48:34.034313 26423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:48:34.034812 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26423
I20260806 10:48:34.035025 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:48:34.035872 16468 external_mini_cluster.cc:1428] Setting key 2a17b9c20d5d49a6ee78c23a95de1824
I20260806 10:48:34.038542 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:34.043955 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:34.037858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.16.21.62:42161)
0806 10:48:34.038883 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:34.038894 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:34.038932 (+ 38us) server_negotiation.cc:407] Connection header received
0806 10:48:34.040550 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:34.040570 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:34.040950 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:34.041331 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:34.042383 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:34.042389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:34.042400 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:34.042414 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:34.042488 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:34.042503 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:34.042739 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:34.042881 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:48:34.043107 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":377,"threads_started":1}
I20260806 10:48:34.049789 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86: Bootstrap starting.
I20260806 10:48:34.052996 26490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:34.053977 26490 log.cc:824] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:34.056784 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86: No bootstrap required, opened a new log
I20260806 10:48:34.063964 26490 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75176fcd4054b3a9ff5bb7653184a86" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42161 } }
I20260806 10:48:34.064375 26490 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:34.064486 26490 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75176fcd4054b3a9ff5bb7653184a86, State: Initialized, Role: FOLLOWER
I20260806 10:48:34.065002 26490 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75176fcd4054b3a9ff5bb7653184a86" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42161 } }
I20260806 10:48:34.065258 26490 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:48:34.065413 26490 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:48:34.065587 26490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:34.067693 26490 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75176fcd4054b3a9ff5bb7653184a86" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42161 } }
I20260806 10:48:34.068145 26490 leader_election.cc:302] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75176fcd4054b3a9ff5bb7653184a86; no voters:
I20260806 10:48:34.068928 26490 leader_election.cc:288] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:48:34.069099 26495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:34.070122 26495 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [term 1 LEADER]: Becoming Leader. State: Replica: f75176fcd4054b3a9ff5bb7653184a86, State: Running, Role: LEADER
I20260806 10:48:34.070621 26495 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75176fcd4054b3a9ff5bb7653184a86" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42161 } }
I20260806 10:48:34.071208 26490 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:48:34.073136 26496 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75176fcd4054b3a9ff5bb7653184a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75176fcd4054b3a9ff5bb7653184a86" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42161 } } }
I20260806 10:48:34.073508 26496 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:34.074172 26497 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75176fcd4054b3a9ff5bb7653184a86. Latest consensus state: current_term: 1 leader_uuid: "f75176fcd4054b3a9ff5bb7653184a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75176fcd4054b3a9ff5bb7653184a86" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 42161 } } }
I20260806 10:48:34.074611 26497 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86 [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:34.075073 26501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:34.080564 26501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:34.088474 26501 catalog_manager.cc:1347] Generated new cluster ID: 2d7912f6a5794e9b9557e2bd900b1a24
I20260806 10:48:34.088616 26501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:34.111338 26501 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:48:34.111778 26501 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:34.118021 26501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75176fcd4054b3a9ff5bb7653184a86: Generated new TSK 0
I20260806 10:48:34.118712 26501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:48:34.128711 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42161
--builtin_ntp_servers=127.16.21.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 10:48:34.462958 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:34.463205 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:34.463290 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:34.474040 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:34.474164 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:34.474377 26514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:48:34.484032 26514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:34.485347 26514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:34.487134 26514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:34.500083 26520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:34.500036 26521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:34.500000 26523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:34.500913 26514 server_base.cc:1034] running on GCE node
I20260806 10:48:34.501511 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:34.502360 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:34.503543 26514 hybrid_clock.cc:648] HybridClock initialized: now 1786013314503493 us; error 49 us; skew 500 ppm
I20260806 10:48:34.503795 26514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:34.506002 26514 webserver.cc:460] Webserver started at http://127.16.21.1:38329/ using document root <none> and password file <none>
I20260806 10:48:34.506639 26514 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:34.506722 26514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:34.506973 26514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:34.509605 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "fe00ddd30f7648b4902fa7e1d6e58a42"
format_stamp: "Formatted at 2026-08-06 10:48:34 on dist-test-slave-0b58"
server_key: "b81a56996b20cff5e285f03f004d4c3d"
server_key_iv: "0081e61f8a8948f1fc4c64e5949a5d99"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.510308 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "fe00ddd30f7648b4902fa7e1d6e58a42"
format_stamp: "Formatted at 2026-08-06 10:48:34 on dist-test-slave-0b58"
server_key: "b81a56996b20cff5e285f03f004d4c3d"
server_key_iv: "0081e61f8a8948f1fc4c64e5949a5d99"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.515091 26514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:48:34.518332 26529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:34.519543 26514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:48:34.519661 26514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "fe00ddd30f7648b4902fa7e1d6e58a42"
format_stamp: "Formatted at 2026-08-06 10:48:34 on dist-test-slave-0b58"
server_key: "b81a56996b20cff5e285f03f004d4c3d"
server_key_iv: "0081e61f8a8948f1fc4c64e5949a5d99"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.519826 26514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:34.525031 26514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:34.525684 26514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:34.525959 26514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:34.526786 26514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:34.528052 26514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:34.528142 26514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:34.528218 26514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:34.528324 26514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:34.561304 26514 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:35491
I20260806 10:48:34.561375 26641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:35491 every 8 connection(s)
I20260806 10:48:34.562620 26514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:48:34.566800 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26514
I20260806 10:48:34.567027 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:48:34.567867 16468 external_mini_cluster.cc:1428] Setting key 92307cb3410ae5dfc8afda152a676617
I20260806 10:48:34.571116 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42161
--builtin_ntp_servers=127.16.21.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 10:48:34.572948 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:34.565048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:49697 (local address 127.16.21.62:42161)
0806 10:48:34.565534 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:34.565540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:34.566804 (+ 1264us) server_negotiation.cc:407] Connection header received
0806 10:48:34.569234 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:34.569240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:34.569298 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:34.569423 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:34.571950 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:34.571959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:34.571968 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:34.571992 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:34.572054 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:34.572098 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:34.572225 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:34.572503 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 10:48:34.572573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260806 10:48:34.573699 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:34.565620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:42161 (local address 127.16.21.1:49697)
0806 10:48:34.566611 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:34.566670 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:34.568949 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:34.569604 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:34.569612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:34.569695 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:34.571655 (+ 1960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:48:34.571658 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:34.571703 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:48:34.571787 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:34.572218 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:34.572228 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:34.572234 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 10:48:34.572536 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 10:48:34.572783 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":263,"threads_started":1}
I20260806 10:48:34.575289 26642 heartbeater.cc:347] Connected to a master server at 127.16.21.62:42161
I20260806 10:48:34.575582 26642 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:34.576188 26642 heartbeater.cc:511] Master 127.16.21.62:42161 requested a full tablet report, sending...
I20260806 10:48:34.578214 26455 ts_manager.cc:194] Registered new tserver with Master: fe00ddd30f7648b4902fa7e1d6e58a42 (127.16.21.1:35491)
W20260806 10:48:34.901257 26646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:34.901508 26646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:34.901675 26646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:34.908839 26646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:34.908950 26646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:34.909121 26646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:48:34.917843 26646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:34.919090 26646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:34.920871 26646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:34.934356 26654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:34.934386 26652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:34.934437 26651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:34.940209 26646 server_base.cc:1034] running on GCE node
I20260806 10:48:34.940933 26646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:34.941921 26646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:34.943152 26646 hybrid_clock.cc:648] HybridClock initialized: now 1786013314943111 us; error 60 us; skew 500 ppm
I20260806 10:48:34.943485 26646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:34.945744 26646 webserver.cc:460] Webserver started at http://127.16.21.2:36937/ using document root <none> and password file <none>
I20260806 10:48:34.946465 26646 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:34.946600 26646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:34.946933 26646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:34.949882 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "771adef3e4644334ae511742596574d6"
format_stamp: "Formatted at 2026-08-06 10:48:34 on dist-test-slave-0b58"
server_key: "69804b71257944b8c70ab60f45948dcf"
server_key_iv: "fe172f18312bcb28a82e3e0cf23dbf12"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.950680 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "771adef3e4644334ae511742596574d6"
format_stamp: "Formatted at 2026-08-06 10:48:34 on dist-test-slave-0b58"
server_key: "69804b71257944b8c70ab60f45948dcf"
server_key_iv: "fe172f18312bcb28a82e3e0cf23dbf12"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.955694 26646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 10:48:34.959276 26660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:34.960765 26646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:34.960933 26646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "771adef3e4644334ae511742596574d6"
format_stamp: "Formatted at 2026-08-06 10:48:34 on dist-test-slave-0b58"
server_key: "69804b71257944b8c70ab60f45948dcf"
server_key_iv: "fe172f18312bcb28a82e3e0cf23dbf12"
server_key_version: "encryptionkey@0"
I20260806 10:48:34.961189 26646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:34.966768 26646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:34.967507 26646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:34.967779 26646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:34.968792 26646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:34.970175 26646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:34.970290 26646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:34.970418 26646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:34.970515 26646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:35.004106 26646 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:46143
I20260806 10:48:35.004199 26775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:46143 every 8 connection(s)
I20260806 10:48:35.005575 26646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:48:35.010154 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26646
I20260806 10:48:35.010354 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:48:35.010808 16468 external_mini_cluster.cc:1428] Setting key 43aa615b0f536e92ed209c256fbea7e5
I20260806 10:48:35.013953 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42161
--builtin_ntp_servers=127.16.21.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 10:48:35.014074 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.008210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:46423 (local address 127.16.21.62:42161)
0806 10:48:35.008459 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:35.008465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:35.009859 (+ 1394us) server_negotiation.cc:407] Connection header received
0806 10:48:35.011716 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:35.011720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:35.011799 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:35.011916 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:35.013319 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:35.013325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:35.013329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:35.013347 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:35.013388 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:35.013405 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:35.013529 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:35.013705 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 10:48:35.013774 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 10:48:35.014866 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.008770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:42161 (local address 127.16.21.2:46423)
0806 10:48:35.009682 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:35.009717 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:35.011424 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:35.012153 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:35.012161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:35.012245 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:35.013019 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:48:35.013022 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:35.013058 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:48:35.013140 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:35.013507 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:35.013514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:35.013520 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 10:48:35.013724 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 10:48:35.013910 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":264,"threads_started":1}
I20260806 10:48:35.016156 26776 heartbeater.cc:347] Connected to a master server at 127.16.21.62:42161
I20260806 10:48:35.016500 26776 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:35.017133 26776 heartbeater.cc:511] Master 127.16.21.62:42161 requested a full tablet report, sending...
I20260806 10:48:35.018642 26455 ts_manager.cc:194] Registered new tserver with Master: 771adef3e4644334ae511742596574d6 (127.16.21.2:46143)
W20260806 10:48:35.349232 26779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:35.349489 26779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:35.349668 26779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:35.356853 26779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:35.356940 26779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:35.357142 26779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:48:35.366124 26779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:35.367308 26779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:35.369055 26779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:35.382124 26786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:35.382056 26788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:35.382138 26785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:35.383347 26779 server_base.cc:1034] running on GCE node
I20260806 10:48:35.383837 26779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:35.384688 26779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:35.385892 26779 hybrid_clock.cc:648] HybridClock initialized: now 1786013315385855 us; error 31 us; skew 500 ppm
I20260806 10:48:35.386142 26779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:48:35.392020 26779 webserver.cc:460] Webserver started at http://127.16.21.3:46527/ using document root <none> and password file <none>
I20260806 10:48:35.392726 26779 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:35.392820 26779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:35.393078 26779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:35.395726 26779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "355f7ebf96934625bb68f016d4fc40a9"
format_stamp: "Formatted at 2026-08-06 10:48:35 on dist-test-slave-0b58"
server_key: "3751924378ab2b11d1340e0168ee0785"
server_key_iv: "0a12c115cf3f8aae54f5ad07eb70f736"
server_key_version: "encryptionkey@0"
I20260806 10:48:35.396492 26779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "355f7ebf96934625bb68f016d4fc40a9"
format_stamp: "Formatted at 2026-08-06 10:48:35 on dist-test-slave-0b58"
server_key: "3751924378ab2b11d1340e0168ee0785"
server_key_iv: "0a12c115cf3f8aae54f5ad07eb70f736"
server_key_version: "encryptionkey@0"
I20260806 10:48:35.401291 26779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:48:35.404345 26794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:35.405685 26779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:48:35.405867 26779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "355f7ebf96934625bb68f016d4fc40a9"
format_stamp: "Formatted at 2026-08-06 10:48:35 on dist-test-slave-0b58"
server_key: "3751924378ab2b11d1340e0168ee0785"
server_key_iv: "0a12c115cf3f8aae54f5ad07eb70f736"
server_key_version: "encryptionkey@0"
I20260806 10:48:35.406174 26779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:35.411615 26779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:35.412542 26779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:35.412812 26779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:35.413694 26779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:35.415005 26779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:35.415129 26779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:35.415270 26779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:35.415364 26779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:35.448493 26779 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:38775
I20260806 10:48:35.448561 26906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:38775 every 8 connection(s)
I20260806 10:48:35.449749 26779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:48:35.453032 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26779
I20260806 10:48:35.453227 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:48:35.453666 16468 external_mini_cluster.cc:1428] Setting key 1d7bb8695281013bfb1e242b42c42daf
I20260806 10:48:35.457970 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.452350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:39649 (local address 127.16.21.62:42161)
0806 10:48:35.452529 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:35.452547 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:35.454012 (+ 1465us) server_negotiation.cc:407] Connection header received
0806 10:48:35.455919 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:35.455924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:35.455988 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:35.456096 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:35.457377 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:35.457381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:35.457384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:35.457399 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:35.457434 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:35.457445 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:35.457543 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:35.457710 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 10:48:35.457757 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:48:35.458788 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.452838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:42161 (local address 127.16.21.3:39649)
0806 10:48:35.453834 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:35.453883 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:35.455669 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:35.456313 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:35.456321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:35.456402 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:35.457117 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:48:35.457120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:35.457153 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:48:35.457230 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:35.457527 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:35.457534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:35.457539 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:48:35.457752 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 10:48:35.457924 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":330,"threads_started":1}
I20260806 10:48:35.460167 26907 heartbeater.cc:347] Connected to a master server at 127.16.21.62:42161
I20260806 10:48:35.460479 26907 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:35.460992 26907 heartbeater.cc:511] Master 127.16.21.62:42161 requested a full tablet report, sending...
I20260806 10:48:35.462316 26455 ts_manager.cc:194] Registered new tserver with Master: 355f7ebf96934625bb68f016d4fc40a9 (127.16.21.3:38775)
I20260806 10:48:35.469408 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 10:48:35.478505 26644 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:48:35.475879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.16.21.62:42161)
0806 10:48:35.477562 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:35.477568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:35.477587 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:48:35.477644 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:35.477650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:35.477707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:35.477822 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:35.478257 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47004: BlockingRecv error: recv got EOF from 127.0.0.1:47004 (error 108)
Metrics: {"server-negotiator.queue_time_us":1558}
I20260806 10:48:35.479509 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26514
I20260806 10:48:35.484050 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.480568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.16.21.1:35491)
0806 10:48:35.481060 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:35.481066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:35.481090 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:48:35.481146 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:35.481151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:35.481385 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:35.481562 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:35.483134 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:35.483146 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:35.483161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:35.483184 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:35.483257 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:35.483285 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:35.483437 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:35.483579 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:48:35.483704 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260806 10:48:35.485256 26637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:35.574275 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26514
I20260806 10:48:35.593668 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26646
I20260806 10:48:35.597203 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.594606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.16.21.2:46143)
0806 10:48:35.595096 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:35.595102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:35.595128 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:48:35.595189 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:35.595195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:35.595418 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:35.595576 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:35.596505 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:35.596511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:35.596521 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:35.596533 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:35.596577 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:35.596592 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:35.596728 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:35.596843 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 10:48:35.596952 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":209,"threads_started":1}
I20260806 10:48:35.598369 26771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:35.678781 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26646
I20260806 10:48:35.697446 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26779
I20260806 10:48:35.701108 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:35.698373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.16.21.3:38775)
0806 10:48:35.699008 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:35.699015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:35.699044 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:48:35.699116 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:35.699122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:35.699324 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:35.699441 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:35.700500 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:35.700506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:35.700517 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:48:35.700529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:35.700568 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:48:35.700581 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:35.700709 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:35.700766 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 10:48:35.700842 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":327,"threads_started":1}
I20260806 10:48:35.702373 26902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:35.782847 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26779
I20260806 10:48:35.801328 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26423
I20260806 10:48:35.802523 26485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:35.907680 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26423
2026-08-06T10:48:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2359 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:48:36 dist-test-slave-0b58 krb5kdc[26925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:48:36 dist-test-slave-0b58 krb5kdc[26925](info): set up 2 sockets
Aug 06 10:48:36 dist-test-slave-0b58 krb5kdc[26925](info): commencing operation
krb5kdc: starting...
W20260806 10:48:37.028956 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:48:37 dist-test-slave-0b58 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:48:37.299346 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46159
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:46159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:37.633059 26938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:37.633390 26938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:37.633562 26938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:37.640841 26938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:48:37.640944 26938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:37.641024 26938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:37.641134 26938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:48:37.641227 26938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:48:37.651144 26938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:46159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:37.652504 26938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:37.653855 26938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:37.665627 26943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:37.665575 26944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:37.665799 26946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:37.666744 26938 server_base.cc:1034] running on GCE node
I20260806 10:48:37.667487 26938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:37.668901 26938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:37.670112 26938 hybrid_clock.cc:648] HybridClock initialized: now 1786013317670073 us; error 27 us; skew 500 ppm
I20260806 10:48:37.670478 26938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:37 dist-test-slave-0b58 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013317, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:37.678313 26938 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:48:37.679872 26938 webserver.cc:460] Webserver started at http://127.16.21.62:46423/ using document root <none> and password file <none>
I20260806 10:48:37.680656 26938 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:37.680768 26938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:37.681023 26938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:37.684042 26938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "3ec0ef3a168f41f7bde817455947dbdf"
format_stamp: "Formatted at 2026-08-06 10:48:37 on dist-test-slave-0b58"
server_key: "f5388435f5a81553ca19a5573d92346c"
server_key_iv: "43691cd2553c5763cb87e0315e934bf6"
server_key_version: "encryptionkey@0"
I20260806 10:48:37.684839 26938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "3ec0ef3a168f41f7bde817455947dbdf"
format_stamp: "Formatted at 2026-08-06 10:48:37 on dist-test-slave-0b58"
server_key: "f5388435f5a81553ca19a5573d92346c"
server_key_iv: "43691cd2553c5763cb87e0315e934bf6"
server_key_version: "encryptionkey@0"
I20260806 10:48:37.689947 26938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:48:37.693279 26953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:37.694561 26938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:48:37.694737 26938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "3ec0ef3a168f41f7bde817455947dbdf"
format_stamp: "Formatted at 2026-08-06 10:48:37 on dist-test-slave-0b58"
server_key: "f5388435f5a81553ca19a5573d92346c"
server_key_iv: "43691cd2553c5763cb87e0315e934bf6"
server_key_version: "encryptionkey@0"
I20260806 10:48:37.694937 26938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:37.707736 26938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:37.711635 26938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:37.711943 26938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:37.731110 26938 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:46159
I20260806 10:48:37.731142 27006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:46159 every 8 connection(s)
I20260806 10:48:37.732553 26938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:48:37.735404 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 26938
I20260806 10:48:37.735656 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:48:37.736518 16468 external_mini_cluster.cc:1428] Setting key df12ae1fdf823f79e0338f7d17b81e46
I20260806 10:48:37.736712 27007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:37.751528 27007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: Bootstrap starting.
Aug 06 10:48:37 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:37.755474 27007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:37.756846 27007 log.cc:824] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:37.760329 27007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: No bootstrap required, opened a new log
I20260806 10:48:37.761564 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:37.738587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.16.21.62:46159)
0806 10:48:37.739375 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:37.739386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:37.739427 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:48:37.741447 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:37.741469 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:37.741825 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:37.742123 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:37.744012 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:37.745392 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:37.747245 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:37.747940 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:37.748065 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:37.753064 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:37.753076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:37.753101 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:37.753158 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:37.757137 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:37.757943 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:37.757949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:37.757955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:37.758130 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:37.758528 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:37.758532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:37.758535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:37.759211 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:37.759513 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:37.759807 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 10:48:37.760207 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":375,"threads_started":1}
I20260806 10:48:37.769352 27007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec0ef3a168f41f7bde817455947dbdf" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46159 } }
I20260806 10:48:37.769697 27007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:37.769771 27007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec0ef3a168f41f7bde817455947dbdf, State: Initialized, Role: FOLLOWER
I20260806 10:48:37.770325 27007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec0ef3a168f41f7bde817455947dbdf" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46159 } }
I20260806 10:48:37.770529 27007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:48:37.770622 27007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:48:37.770740 27007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:37.773099 27007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec0ef3a168f41f7bde817455947dbdf" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46159 } }
I20260806 10:48:37.773522 27007 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ec0ef3a168f41f7bde817455947dbdf; no voters:
I20260806 10:48:37.774062 27007 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:48:37.774227 27012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:37.774899 27012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [term 1 LEADER]: Becoming Leader. State: Replica: 3ec0ef3a168f41f7bde817455947dbdf, State: Running, Role: LEADER
I20260806 10:48:37.775457 27012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec0ef3a168f41f7bde817455947dbdf" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46159 } }
I20260806 10:48:37.776153 27007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:48:37.778748 27013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ec0ef3a168f41f7bde817455947dbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec0ef3a168f41f7bde817455947dbdf" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46159 } } }
I20260806 10:48:37.778811 27014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ec0ef3a168f41f7bde817455947dbdf. Latest consensus state: current_term: 1 leader_uuid: "3ec0ef3a168f41f7bde817455947dbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec0ef3a168f41f7bde817455947dbdf" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46159 } } }
I20260806 10:48:37.779228 27013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [sys.catalog]: This master's current role is: LEADER
I20260806 10:48:37.779225 27014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf [sys.catalog]: This master's current role is: LEADER
W20260806 10:48:37.787736 27028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:48:37.787848 27028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:48:37.788003 27019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:48:37.790328 27019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:48:37.798128 27019 catalog_manager.cc:1347] Generated new cluster ID: fe357121ebef41a8a2dc7fe1a27ed926
I20260806 10:48:37.798230 27019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:48:37.810299 27019 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:48:37.812044 27019 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:48:37.827080 27019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: Generated new TSK 0
I20260806 10:48:37.828148 27019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:48:37.904016 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46159
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:38.241259 27033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:38.241549 27033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:38.241740 27033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:38.248813 27033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:38.248934 27033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:38.249109 27033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:48:38.258293 27033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:38.259467 27033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:38.261044 27033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:38.273883 27039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:38.273962 27041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:38.273972 27038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:38.274770 27033 server_base.cc:1034] running on GCE node
I20260806 10:48:38.275270 27033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:38.276197 27033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:38.277436 27033 hybrid_clock.cc:648] HybridClock initialized: now 1786013318277386 us; error 42 us; skew 500 ppm
I20260806 10:48:38.277742 27033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:38 dist-test-slave-0b58 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013318, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:38.281170 27033 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:48:38.282589 27033 webserver.cc:460] Webserver started at http://127.16.21.1:34481/ using document root <none> and password file <none>
I20260806 10:48:38.283329 27033 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:38.283481 27033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:38.283772 27033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:38.286558 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "f016e024cb2d4d0c83340f7a228f4343"
format_stamp: "Formatted at 2026-08-06 10:48:38 on dist-test-slave-0b58"
server_key: "36f3bec9f5ec52bf24f5d8ab87d9d2f7"
server_key_iv: "bd7abf519573de2d984ca90954d6f69f"
server_key_version: "encryptionkey@0"
I20260806 10:48:38.287309 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "f016e024cb2d4d0c83340f7a228f4343"
format_stamp: "Formatted at 2026-08-06 10:48:38 on dist-test-slave-0b58"
server_key: "36f3bec9f5ec52bf24f5d8ab87d9d2f7"
server_key_iv: "bd7abf519573de2d984ca90954d6f69f"
server_key_version: "encryptionkey@0"
I20260806 10:48:38.292372 27033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 10:48:38.295562 27048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:38.296834 27033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:48:38.297003 27033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "f016e024cb2d4d0c83340f7a228f4343"
format_stamp: "Formatted at 2026-08-06 10:48:38 on dist-test-slave-0b58"
server_key: "36f3bec9f5ec52bf24f5d8ab87d9d2f7"
server_key_iv: "bd7abf519573de2d984ca90954d6f69f"
server_key_version: "encryptionkey@0"
I20260806 10:48:38.297190 27033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:38.320778 27033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:38.324450 27033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:38.324725 27033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:38.325599 27033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:38.327181 27033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:38.327289 27033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:38.327423 27033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:38.327476 27033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:38.360479 27033 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:35663
I20260806 10:48:38.360525 27162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:35663 every 8 connection(s)
I20260806 10:48:38.361716 27033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:48:38.362406 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 27033
I20260806 10:48:38.362675 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:48:38.363446 16468 external_mini_cluster.cc:1428] Setting key 1cd994e3dfc678950edff281adf3f8dd
Aug 06 10:48:38 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013318, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:38.385351 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:38.364600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:36931 (local address 127.16.21.62:46159)
0806 10:48:38.365097 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:38.365103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:38.366210 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 10:48:38.369130 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:38.369134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:38.369196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:38.369299 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:38.371358 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.372063 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.374527 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.374935 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.375033 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:38.379169 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:38.379174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:38.379177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:38.379207 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:38.382196 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:38.382908 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:38.382913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:38.382916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:38.383036 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:38.383600 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:38.383606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:38.383609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:38.383878 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:38.384032 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:38.384781 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 10:48:38.385000 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260806 10:48:38.386234 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:38.365177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46159 (local address 127.16.21.1:36931)
0806 10:48:38.366024 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:38.366075 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:38.368856 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:38.369549 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:38.369561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:38.370319 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:38.371086 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:38.371113 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.372237 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.372246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:38.374313 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:38.374330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.375169 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.375179 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:38.375510 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:38.376294 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:38.376327 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:38.378921 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:38.382343 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:38.382352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:38.382366 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:38.382771 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:38.383145 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:38.383150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:38.383153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:38.383478 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:38.384015 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:38.384022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:38.384407 (+ 385us) client_negotiation.cc:769] Sending connection context
0806 10:48:38.384668 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 10:48:38.385139 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":340,"threads_started":1}
I20260806 10:48:38.387702 27163 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46159
I20260806 10:48:38.388033 27163 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:38.388913 27163 heartbeater.cc:511] Master 127.16.21.62:46159 requested a full tablet report, sending...
I20260806 10:48:38.391057 26971 ts_manager.cc:194] Registered new tserver with Master: f016e024cb2d4d0c83340f7a228f4343 (127.16.21.1:35663)
I20260806 10:48:38.392552 26971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:36931
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:48:38.431525 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46159
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:38.764621 27169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:38.764885 27169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:38.764976 27169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:38.772444 27169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:38.772536 27169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:38.772665 27169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:48:38.782115 27169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:38.783280 27169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:38.784606 27169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:38.797261 27174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:38.797240 27177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:38.797119 27175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:38.797943 27169 server_base.cc:1034] running on GCE node
I20260806 10:48:38.798460 27169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:38.799310 27169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:38.800494 27169 hybrid_clock.cc:648] HybridClock initialized: now 1786013318800443 us; error 45 us; skew 500 ppm
I20260806 10:48:38.800822 27169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:38 dist-test-slave-0b58 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013318, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:38.803704 27169 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:48:38.805130 27169 webserver.cc:460] Webserver started at http://127.16.21.2:44689/ using document root <none> and password file <none>
I20260806 10:48:38.805810 27169 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:38.805944 27169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:38.806288 27169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:38.809126 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "a8b72327cab84b6db372166c56ab1d57"
format_stamp: "Formatted at 2026-08-06 10:48:38 on dist-test-slave-0b58"
server_key: "6743f47f30457dd811a65f5c9eccc17d"
server_key_iv: "649c9afaae9905da0e82c9107dd5598f"
server_key_version: "encryptionkey@0"
I20260806 10:48:38.809882 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "a8b72327cab84b6db372166c56ab1d57"
format_stamp: "Formatted at 2026-08-06 10:48:38 on dist-test-slave-0b58"
server_key: "6743f47f30457dd811a65f5c9eccc17d"
server_key_iv: "649c9afaae9905da0e82c9107dd5598f"
server_key_version: "encryptionkey@0"
I20260806 10:48:38.814718 27169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:48:38.818010 27184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:38.819190 27169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 10:48:38.819375 27169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "a8b72327cab84b6db372166c56ab1d57"
format_stamp: "Formatted at 2026-08-06 10:48:38 on dist-test-slave-0b58"
server_key: "6743f47f30457dd811a65f5c9eccc17d"
server_key_iv: "649c9afaae9905da0e82c9107dd5598f"
server_key_version: "encryptionkey@0"
I20260806 10:48:38.819566 27169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:38.843152 27169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:38.846916 27169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:38.847208 27169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:38.848155 27169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:38.849473 27169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:38.849577 27169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:38.849717 27169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:38.849795 27169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:38.883167 27169 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:39287
I20260806 10:48:38.883267 27298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:39287 every 8 connection(s)
I20260806 10:48:38.884467 27169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:48:38.886015 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 27169
I20260806 10:48:38.886241 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:48:38.886816 16468 external_mini_cluster.cc:1428] Setting key 4d69de551a6f57f23b8c7576b4e6eb57
Aug 06 10:48:38 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013318, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:38.904356 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:38.887083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:38291 (local address 127.16.21.62:46159)
0806 10:48:38.887582 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:38.887608 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:38.888753 (+ 1145us) server_negotiation.cc:407] Connection header received
0806 10:48:38.890568 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:38.890573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:38.890653 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:38.890772 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:38.892282 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.893025 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.894485 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.894907 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.895041 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:38.898525 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:38.898530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:38.898533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:38.898564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:38.901261 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:38.902076 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:38.902083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:38.902086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:38.902214 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:38.902639 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:38.902645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:38.902648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:38.902926 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:38.903086 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:38.903719 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 10:48:38.903937 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260806 10:48:38.905282 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:38.887699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46159 (local address 127.16.21.2:38291)
0806 10:48:38.888589 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:38.888641 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:38.890299 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:38.891022 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:38.891030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:38.891523 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:38.892080 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:38.892097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.893184 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.893190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:38.894338 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:38.894349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:38.895134 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:38.895143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:38.895376 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:38.896045 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:38.896066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:38.898310 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:38.901459 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:38.901474 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:38.901485 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:38.901956 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:38.902333 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:38.902338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:38.902342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:38.902524 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:38.903067 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:38.903074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:38.903381 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 10:48:38.903640 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 10:48:38.904147 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":353,"threads_started":1}
I20260806 10:48:38.906751 27299 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46159
I20260806 10:48:38.907066 27299 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:38.908038 27299 heartbeater.cc:511] Master 127.16.21.62:46159 requested a full tablet report, sending...
I20260806 10:48:38.909626 26971 ts_manager.cc:194] Registered new tserver with Master: a8b72327cab84b6db372166c56ab1d57 (127.16.21.2:39287)
I20260806 10:48:38.910784 26971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:38291
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:48:38.952603 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46159
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:48:39.282241 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:48:39.282480 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:48:39.282635 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:48:39.289669 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:48:39.289762 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:48:39.289952 27305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:48:39.299149 27305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:48:39.300374 27305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:48:39.301615 27305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:48:39.313962 27310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:39.314002 27311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:48:39.314029 27313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:48:39.314803 27305 server_base.cc:1034] running on GCE node
I20260806 10:48:39.315339 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:48:39.316110 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:48:39.317343 27305 hybrid_clock.cc:648] HybridClock initialized: now 1786013319317302 us; error 62 us; skew 500 ppm
I20260806 10:48:39.317754 27305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:48:39 dist-test-slave-0b58 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:48:39.321038 27305 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:48:39.322463 27305 webserver.cc:460] Webserver started at http://127.16.21.3:44235/ using document root <none> and password file <none>
I20260806 10:48:39.323150 27305 fs_manager.cc:359] Metadata directory not provided
I20260806 10:48:39.323292 27305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:48:39.323622 27305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:48:39.326464 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "52c540c47846474693ae0e870f544101"
format_stamp: "Formatted at 2026-08-06 10:48:39 on dist-test-slave-0b58"
server_key: "e54ee6f7b0b517005f4aaf9ea24c6d2d"
server_key_iv: "16b897569441f69b8574e6c971b11875"
server_key_version: "encryptionkey@0"
I20260806 10:48:39.327196 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "52c540c47846474693ae0e870f544101"
format_stamp: "Formatted at 2026-08-06 10:48:39 on dist-test-slave-0b58"
server_key: "e54ee6f7b0b517005f4aaf9ea24c6d2d"
server_key_iv: "16b897569441f69b8574e6c971b11875"
server_key_version: "encryptionkey@0"
I20260806 10:48:39.331926 27305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260806 10:48:39.335218 27320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:39.336450 27305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:48:39.336621 27305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "52c540c47846474693ae0e870f544101"
format_stamp: "Formatted at 2026-08-06 10:48:39 on dist-test-slave-0b58"
server_key: "e54ee6f7b0b517005f4aaf9ea24c6d2d"
server_key_iv: "16b897569441f69b8574e6c971b11875"
server_key_version: "encryptionkey@0"
I20260806 10:48:39.336814 27305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:48:39.351773 27305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:48:39.355777 27305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:48:39.356089 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:48:39.357025 27305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:48:39.358263 27305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:48:39.358362 27305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:39.358506 27305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:48:39.358580 27305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:48:39.393052 27305 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:35551
I20260806 10:48:39.393100 27434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:35551 every 8 connection(s)
I20260806 10:48:39.394284 27305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:48:39.395991 27163 heartbeater.cc:503] Master 127.16.21.62:46159 was elected leader, sending a full tablet report...
I20260806 10:48:39.399502 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 27305
I20260806 10:48:39.399693 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:48:39.400170 16468 external_mini_cluster.cc:1428] Setting key cf64ccdd9a9f3d2a756085b488664707
Aug 06 10:48:39 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:48:39.414206 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.396935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:60697 (local address 127.16.21.62:46159)
0806 10:48:39.397133 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.397140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.398376 (+ 1236us) server_negotiation.cc:407] Connection header received
0806 10:48:39.400420 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.400425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.400523 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.400634 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:39.402321 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.403105 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.404684 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.405138 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.405273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.408621 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:39.408627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:39.408632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:39.408671 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:39.411231 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:39.411933 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:39.411938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:39.411940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:39.412061 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:39.412558 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:39.412563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:39.412568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:39.412838 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:39.412989 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.413630 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.413873 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:48:39.415236 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.397383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46159 (local address 127.16.21.3:60697)
0806 10:48:39.398238 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:39.398274 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:39.400066 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:39.400920 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:39.400936 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:39.401423 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:48:39.402150 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.402168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.403270 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.403275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.404539 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.404551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.405372 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.405380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.405621 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.406302 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:48:39.406323 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:48:39.408413 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:48:39.411367 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:39.411375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:39.411388 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:39.411804 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:39.412172 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:48:39.412177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:48:39.412183 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:48:39.412440 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:48:39.412978 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:48:39.412986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:48:39.413275 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 10:48:39.413517 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 10:48:39.414006 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":351,"threads_started":1}
I20260806 10:48:39.416482 27435 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46159
I20260806 10:48:39.416713 27435 heartbeater.cc:464] Registering TS with master...
I20260806 10:48:39.417333 27435 heartbeater.cc:511] Master 127.16.21.62:46159 requested a full tablet report, sending...
I20260806 10:48:39.418581 26971 ts_manager.cc:194] Registered new tserver with Master: 52c540c47846474693ae0e870f544101 (127.16.21.3:35551)
I20260806 10:48:39.419313 26971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:60697
I20260806 10:48:39.427098 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:48:39.443178 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.431397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.16.21.62:46159)
0806 10:48:39.431611 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.431615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.432111 (+ 496us) server_negotiation.cc:407] Connection header received
0806 10:48:39.432238 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.432241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.432367 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.432448 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:39.433783 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.434439 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.435564 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.435939 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.436020 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.438217 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:39.438221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:39.438224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:39.438251 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:39.440881 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:39.441519 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:39.441522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:39.441524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:39.441641 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:39.441980 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:39.441983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:39.441985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:39.442248 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:39.442445 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.442773 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.442932 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 10:48:39.447046 26971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:48:39.454164 26971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:48:39.479208 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.469327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:35551 (local address 127.0.0.1:47406)
0806 10:48:39.471694 (+ 2367us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:39.471711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:39.471848 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:39.472680 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:39.472685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:39.472709 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:39.473149 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.473165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.474541 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.474546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.476230 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.476243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.478077 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.478083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.478311 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.478387 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 10:48:39.478519 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:48:39.478713 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2235,"mutex_wait_us":69,"thread_start_us":2064,"threads_started":1}
I20260806 10:48:39.479260 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.469631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.16.21.3:35551)
0806 10:48:39.471279 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.471288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.471825 (+ 537us) server_negotiation.cc:407] Connection header received
0806 10:48:39.472011 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.472018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.472206 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.472418 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:39.473350 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.474383 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.476438 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.477880 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.478018 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.478253 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.478748 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.478923 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":1284,"threads_started":1}
I20260806 10:48:39.479496 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.465999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.16.21.1:35663)
0806 10:48:39.468610 (+ 2611us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.468620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.468650 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:48:39.469830 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.469839 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.470048 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.470548 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:39.471373 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.473084 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.476723 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.478135 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.478690 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.478929 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.479000 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.479162 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2393,"thread_start_us":2163,"threads_started":1}
I20260806 10:48:39.479846 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.465519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:35663 (local address 127.0.0.1:40222)
0806 10:48:39.467601 (+ 2082us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:39.467646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:39.467813 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:39.470303 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:39.470310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:39.470359 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:39.470864 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.470881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.473251 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.473256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.476361 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.476377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.478370 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.478378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.478584 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.478692 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 10:48:39.478775 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 10:48:39.478926 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1861,"mutex_wait_us":311,"spinlock_wait_cycles":4992,"thread_start_us":3393,"threads_started":1}
I20260806 10:48:39.480166 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.466140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:39287 (local address 127.0.0.1:56650)
0806 10:48:39.469192 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:39.469209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:39.469339 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:39.470308 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:39.470312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:39.470357 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:39.471517 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.471538 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.473305 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.473316 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.476362 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.476377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.478421 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.478427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.478627 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.479441 (+ 814us) client_negotiation.cc:769] Sending connection context
0806 10:48:39.479527 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 10:48:39.479660 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2903,"mutex_wait_us":1034,"spinlock_wait_cycles":8448,"thread_start_us":1071,"threads_started":1}
I20260806 10:48:39.480541 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.466772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.16.21.2:39287)
0806 10:48:39.468976 (+ 2204us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.468987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.469461 (+ 474us) server_negotiation.cc:407] Connection header received
0806 10:48:39.469568 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.469576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.469828 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.470010 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:39.471740 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.473134 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.476551 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.478121 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.478258 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.478469 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.479928 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.480088 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2029,"thread_start_us":1827,"threads_started":1}
I20260806 10:48:39.483346 27370 tablet_service.cc:1467] Processing CreateTablet for tablet fe752666be61431fb9a5ad893c23adf2 (DEFAULT_TABLE table=test-table [id=1d4bf3c65c7e4fe58942082e1272a856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:39.484035 27098 tablet_service.cc:1467] Processing CreateTablet for tablet fe752666be61431fb9a5ad893c23adf2 (DEFAULT_TABLE table=test-table [id=1d4bf3c65c7e4fe58942082e1272a856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:39.484030 27233 tablet_service.cc:1467] Processing CreateTablet for tablet fe752666be61431fb9a5ad893c23adf2 (DEFAULT_TABLE table=test-table [id=1d4bf3c65c7e4fe58942082e1272a856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:48:39.484715 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe752666be61431fb9a5ad893c23adf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:39.485442 27233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe752666be61431fb9a5ad893c23adf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:39.485486 27098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe752666be61431fb9a5ad893c23adf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:48:39.500319 27455 tablet_bootstrap.cc:492] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Bootstrap starting.
I20260806 10:48:39.501189 27453 tablet_bootstrap.cc:492] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Bootstrap starting.
I20260806 10:48:39.501824 27454 tablet_bootstrap.cc:492] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Bootstrap starting.
I20260806 10:48:39.504146 27453 tablet_bootstrap.cc:654] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:39.504434 27455 tablet_bootstrap.cc:654] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:39.505270 27453 log.cc:824] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:39.505518 27454 tablet_bootstrap.cc:654] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Neither blocks nor log segments found. Creating new log.
I20260806 10:48:39.505698 27455 log.cc:824] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:39.506819 27454 log.cc:824] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Log is configured to *not* fsync() on all Append() calls
I20260806 10:48:39.509014 27453 tablet_bootstrap.cc:492] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: No bootstrap required, opened a new log
I20260806 10:48:39.509320 27453 ts_tablet_manager.cc:1397] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 10:48:39.509922 27455 tablet_bootstrap.cc:492] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: No bootstrap required, opened a new log
I20260806 10:48:39.510022 27454 tablet_bootstrap.cc:492] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: No bootstrap required, opened a new log
I20260806 10:48:39.510226 27455 ts_tablet_manager.cc:1397] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260806 10:48:39.510322 27454 ts_tablet_manager.cc:1397] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 10:48:39.517799 27455 raft_consensus.cc:348] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.518186 27455 raft_consensus.cc:374] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:39.518337 27455 raft_consensus.cc:729] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b72327cab84b6db372166c56ab1d57, State: Initialized, Role: FOLLOWER
I20260806 10:48:39.518934 27455 consensus_queue.cc:260] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.520009 27299 heartbeater.cc:503] Master 127.16.21.62:46159 was elected leader, sending a full tablet report...
I20260806 10:48:39.520179 27453 raft_consensus.cc:348] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.520614 27453 raft_consensus.cc:374] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:39.520726 27453 raft_consensus.cc:729] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c540c47846474693ae0e870f544101, State: Initialized, Role: FOLLOWER
I20260806 10:48:39.520750 27455 ts_tablet_manager.cc:1428] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260806 10:48:39.521185 27454 raft_consensus.cc:348] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.521407 27453 consensus_queue.cc:260] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.521633 27454 raft_consensus.cc:374] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:48:39.521749 27454 raft_consensus.cc:729] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f016e024cb2d4d0c83340f7a228f4343, State: Initialized, Role: FOLLOWER
I20260806 10:48:39.522529 27435 heartbeater.cc:503] Master 127.16.21.62:46159 was elected leader, sending a full tablet report...
I20260806 10:48:39.522593 27454 consensus_queue.cc:260] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.523344 27453 ts_tablet_manager.cc:1428] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 10:48:39.524966 27454 ts_tablet_manager.cc:1428] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260806 10:48:39.614181 27164 tablet.cc:2369] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:48:39.636461 27300 tablet.cc:2369] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:48:39.645802 27436 tablet.cc:2369] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:48:39.754608 27461 raft_consensus.cc:482] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:48:39.754889 27461 raft_consensus.cc:504] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.756533 27461 leader_election.cc:288] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c540c47846474693ae0e870f544101 (127.16.21.3:35551), a8b72327cab84b6db372166c56ab1d57 (127.16.21.2:39287)
I20260806 10:48:39.765620 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.756960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:57981 (local address 127.16.21.3:35551)
0806 10:48:39.757155 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.757161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.757937 (+ 776us) server_negotiation.cc:407] Connection header received
0806 10:48:39.758143 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.758149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.758227 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.758324 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:39.759266 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.760368 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.763291 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.764495 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.764639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.764738 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.765243 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.765368 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:48:39.765620 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.756961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:59351 (local address 127.16.21.2:39287)
0806 10:48:39.757155 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.757161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.757934 (+ 773us) server_negotiation.cc:407] Connection header received
0806 10:48:39.758143 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.758149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.758227 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.758324 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:48:39.759258 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.760367 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.763285 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.764496 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.764639 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.764737 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.765243 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.765368 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:48:39.765846 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.756946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:35551 (local address 127.16.21.1:57981)
0806 10:48:39.757806 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:39.757826 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:39.757996 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:39.758359 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:39.758364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:39.758398 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:39.759084 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.759096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.760503 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.760508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.763128 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.763143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.764828 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.764833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.765053 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.765107 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 10:48:39.765203 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 10:48:39.765365 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":267,"spinlock_wait_cycles":30720,"thread_start_us":437,"threads_started":1}
I20260806 10:48:39.766242 27463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.756946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:39287 (local address 127.16.21.1:59351)
0806 10:48:39.757806 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0806 10:48:39.757826 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:48:39.757996 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:48:39.758359 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:48:39.758364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:48:39.758395 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:48:39.759082 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.759096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.760504 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.760508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.763128 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:48:39.763143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.764800 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.764813 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:48:39.765059 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.765107 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 10:48:39.765203 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 10:48:39.765365 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":272,"spinlock_wait_cycles":17280,"thread_start_us":394,"threads_started":1}
I20260806 10:48:39.766939 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe752666be61431fb9a5ad893c23adf2" candidate_uuid: "f016e024cb2d4d0c83340f7a228f4343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c540c47846474693ae0e870f544101" is_pre_election: true
I20260806 10:48:39.767171 27253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe752666be61431fb9a5ad893c23adf2" candidate_uuid: "f016e024cb2d4d0c83340f7a228f4343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b72327cab84b6db372166c56ab1d57" is_pre_election: true
I20260806 10:48:39.767424 27390 raft_consensus.cc:2454] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f016e024cb2d4d0c83340f7a228f4343 in term 0.
I20260806 10:48:39.767724 27253 raft_consensus.cc:2454] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f016e024cb2d4d0c83340f7a228f4343 in term 0.
I20260806 10:48:39.768215 27051 leader_election.cc:302] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c540c47846474693ae0e870f544101, f016e024cb2d4d0c83340f7a228f4343; no voters:
I20260806 10:48:39.768666 27461 raft_consensus.cc:2781] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:48:39.768841 27461 raft_consensus.cc:482] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:48:39.768951 27461 raft_consensus.cc:3037] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:39.771484 27461 raft_consensus.cc:504] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.772105 27461 leader_election.cc:288] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [CANDIDATE]: Term 1 election: Requested vote from peers 52c540c47846474693ae0e870f544101 (127.16.21.3:35551), a8b72327cab84b6db372166c56ab1d57 (127.16.21.2:39287)
I20260806 10:48:39.772487 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe752666be61431fb9a5ad893c23adf2" candidate_uuid: "f016e024cb2d4d0c83340f7a228f4343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c540c47846474693ae0e870f544101"
I20260806 10:48:39.772624 27253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe752666be61431fb9a5ad893c23adf2" candidate_uuid: "f016e024cb2d4d0c83340f7a228f4343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b72327cab84b6db372166c56ab1d57"
I20260806 10:48:39.772679 27390 raft_consensus.cc:3037] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:39.772836 27253 raft_consensus.cc:3037] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:48:39.774921 27390 raft_consensus.cc:2454] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f016e024cb2d4d0c83340f7a228f4343 in term 1.
I20260806 10:48:39.775050 27253 raft_consensus.cc:2454] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f016e024cb2d4d0c83340f7a228f4343 in term 1.
I20260806 10:48:39.775296 27051 leader_election.cc:302] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c540c47846474693ae0e870f544101, f016e024cb2d4d0c83340f7a228f4343; no voters:
I20260806 10:48:39.775583 27461 raft_consensus.cc:2781] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:48:39.776063 27461 raft_consensus.cc:686] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 1 LEADER]: Becoming Leader. State: Replica: f016e024cb2d4d0c83340f7a228f4343, State: Running, Role: LEADER
I20260806 10:48:39.776644 27461 consensus_queue.cc:237] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } }
I20260806 10:48:39.781057 26970 catalog_manager.cc:5495] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 reported cstate change: term changed from 0 to 1, leader changed from <none> to f016e024cb2d4d0c83340f7a228f4343 (127.16.21.1). New cstate: current_term: 1 leader_uuid: "f016e024cb2d4d0c83340f7a228f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f016e024cb2d4d0c83340f7a228f4343" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260806 10:48:39.853586 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.848492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.16.21.1:35663)
0806 10:48:39.848694 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.848698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.848716 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:48:39.848881 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.848898 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.848981 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.849101 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:48:39.849436 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.850118 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.851364 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.851774 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.851867 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.852415 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:48:39.852997 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:48:39.853102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.853216 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.853288 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:48:39.861305 27390 raft_consensus.cc:1261] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 1 FOLLOWER]: Refusing update from remote peer f016e024cb2d4d0c83340f7a228f4343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:39.861304 27253 raft_consensus.cc:1261] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 1 FOLLOWER]: Refusing update from remote peer f016e024cb2d4d0c83340f7a228f4343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:48:39.862248 27464 consensus_queue.cc:1035] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8b72327cab84b6db372166c56ab1d57" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:39.862599 27461 consensus_queue.cc:1035] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c540c47846474693ae0e870f544101" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:48:39.873595 27467 mvcc.cc:205] Tried to move back new op lower bound from 7315510558142914560 to 7315510557808750592. Current Snapshot: MvccSnapshot[applied={T|T < 7315510558142914560}]
I20260806 10:48:39.876425 27468 mvcc.cc:205] Tried to move back new op lower bound from 7315510558142914560 to 7315510557808750592. Current Snapshot: MvccSnapshot[applied={T|T < 7315510558142914560}]
I20260806 10:48:39.888598 26970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:39.888876 26970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:48:39.892899 26970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ec0ef3a168f41f7bde817455947dbdf: Sending DeleteTablet for 3 replicas of tablet fe752666be61431fb9a5ad893c23adf2
I20260806 10:48:39.894146 27233 tablet_service.cc:1514] Processing DeleteTablet for tablet fe752666be61431fb9a5ad893c23adf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:39 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:56650
I20260806 10:48:39.894302 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet fe752666be61431fb9a5ad893c23adf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:39 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:47406
I20260806 10:48:39.895140 27098 tablet_service.cc:1514] Processing DeleteTablet for tablet fe752666be61431fb9a5ad893c23adf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:48:39 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:40222
I20260806 10:48:39.902419 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 27033
I20260806 10:48:39.903227 27478 tablet_replica.cc:330] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: stopping tablet replica
I20260806 10:48:39.903735 27478 raft_consensus.cc:2229] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:48:39.904414 27478 raft_consensus.cc:2258] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:39.906968 27478 ts_tablet_manager.cc:1905] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:39.916805 27477 tablet_replica.cc:330] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: stopping tablet replica
I20260806 10:48:39.917282 27477 raft_consensus.cc:2229] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:39.917805 27476 tablet_replica.cc:330] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: stopping tablet replica
I20260806 10:48:39.917790 27477 raft_consensus.cc:2258] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:39.918179 27476 raft_consensus.cc:2229] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:48:39.918637 27476 raft_consensus.cc:2258] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:48:39.918812 27478 ts_tablet_manager.cc:1918] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:39.918982 27478 log.cc:1196] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/fe752666be61431fb9a5ad893c23adf2
I20260806 10:48:39.919544 27478 ts_tablet_manager.cc:1939] T fe752666be61431fb9a5ad893c23adf2 P f016e024cb2d4d0c83340f7a228f4343: Deleting consensus metadata
I20260806 10:48:39.920024 27477 ts_tablet_manager.cc:1905] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:39.920677 27476 ts_tablet_manager.cc:1905] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:48:39.921439 26956 catalog_manager.cc:4841] TS f016e024cb2d4d0c83340f7a228f4343 (127.16.21.1:35663): tablet fe752666be61431fb9a5ad893c23adf2 (table test-table [id=1d4bf3c65c7e4fe58942082e1272a856]) successfully deleted
I20260806 10:48:39.927567 27476 ts_tablet_manager.cc:1918] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:39.927719 27476 log.cc:1196] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/fe752666be61431fb9a5ad893c23adf2
I20260806 10:48:39.928321 27476 ts_tablet_manager.cc:1939] T fe752666be61431fb9a5ad893c23adf2 P a8b72327cab84b6db372166c56ab1d57: Deleting consensus metadata
I20260806 10:48:39.928711 27477 ts_tablet_manager.cc:1918] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:48:39.928901 27477 log.cc:1196] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/fe752666be61431fb9a5ad893c23adf2
I20260806 10:48:39.929467 27477 ts_tablet_manager.cc:1939] T fe752666be61431fb9a5ad893c23adf2 P 52c540c47846474693ae0e870f544101: Deleting consensus metadata
I20260806 10:48:39.929949 26955 catalog_manager.cc:4841] TS a8b72327cab84b6db372166c56ab1d57 (127.16.21.2:39287): tablet fe752666be61431fb9a5ad893c23adf2 (table test-table [id=1d4bf3c65c7e4fe58942082e1272a856]) successfully deleted
I20260806 10:48:39.931116 26956 catalog_manager.cc:4841] TS 52c540c47846474693ae0e870f544101 (127.16.21.3:35551): tablet fe752666be61431fb9a5ad893c23adf2 (table test-table [id=1d4bf3c65c7e4fe58942082e1272a856]) successfully deleted
Aug 06 10:48:39 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:48:39.939508 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:39.903798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.16.21.1:35663)
0806 10:48:39.904246 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:39.904284 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:39.917048 (+ 12764us) server_negotiation.cc:407] Connection header received
0806 10:48:39.917264 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:39.917271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:39.917351 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:39.917455 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:39.922806 (+ 5351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.923851 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.925668 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:39.926394 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:39.926496 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:39.932564 (+ 6068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:39.932583 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:39.932601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:39.932647 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:39.936578 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:39.937294 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:39.937303 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:39.937306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:39.937482 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:39.937866 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:39.937871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:39.937875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:39.938345 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:39.938681 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:39.938860 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 10:48:39.939106 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260806 10:48:39.940295 27158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:40.042662 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 27033
I20260806 10:48:40.061774 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 27169
Aug 06 10:48:40 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:48:40.075999 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:40.062518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.16.21.2:39287)
0806 10:48:40.062691 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:40.062695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:40.062715 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:48:40.062901 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:40.062905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:40.062984 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:40.063088 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:40.064370 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:40.065058 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:40.066269 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:40.066658 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:40.066756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:40.070715 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:40.070728 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:40.070744 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:40.070774 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:40.073633 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:40.074307 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:40.074313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:40.074315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:40.074442 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:40.074761 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:40.074764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:40.074766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:40.075099 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:40.075312 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:40.075545 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 10:48:40.075737 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:48:40.076865 27294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:40.166445 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 27169
I20260806 10:48:40.187398 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 27305
Aug 06 10:48:40 dist-test-slave-0b58 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:48:40.203181 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:48:40.188289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.16.21.3:35551)
0806 10:48:40.188489 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 10:48:40.188495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:48:40.188515 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:48:40.188696 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:48:40.188702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:48:40.188784 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:48:40.188905 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:48:40.190260 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:40.190999 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:40.192286 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:48:40.192746 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:48:40.192885 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:48:40.197270 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:48:40.197288 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:48:40.197304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:48:40.197339 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:48:40.200219 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:40.201121 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:40.201137 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:40.201140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:40.201294 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:48:40.201681 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:48:40.201691 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:48:40.201693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:48:40.202022 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:48:40.202367 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 10:48:40.202630 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 10:48:40.202849 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:48:40.203907 27430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:40.298301 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 27305
I20260806 10:48:40.318231 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 26938
I20260806 10:48:40.319494 27002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:48:40.428860 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 26938
2026-08-06T10:48:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4523 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:48:40 dist-test-slave-0b58 krb5kdc[27486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:48:40 dist-test-slave-0b58 krb5kdc[27486](info): set up 2 sockets
Aug 06 10:48:40 dist-test-slave-0b58 krb5kdc[27486](info): commencing operation
krb5kdc: starting...
W20260806 10:48:41.552754 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.006s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:48:41 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:48:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/rangeradmin_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/rangeradmin_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/rangerlookup_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/rangerlookup_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
I20260806 10:48:41.935546 16468 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLWW7rW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/postgres -l logfile start
2026-08-06 10:48:44.427 UTC [27517] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 10:48:44.427 UTC [27517] LOG: listening on IPv4 address "127.16.21.20", port 53865
2026-08-06 10:48:44.430 UTC [27517] LOG: listening on Unix socket "/tmp/.s.PGSQL.53865"
2026-08-06 10:48:44.454 UTC [27520] LOG: database system was shut down at 2026-08-06 10:48:43 UTC
2026-08-06 10:48:44.457 UTC [27517] LOG: database system is ready to accept connections
I20260806 10:48:45.444546 16468 mini_postgres.cc:96] Postgres bound to 53865
2026-08-06 10:48:45.471 UTC [27528] FATAL: database "slave" does not exist
127.16.21.20:53865 - accepting connections
I20260806 10:48:45.472927 16468 mini_ranger.cc:161] Starting Ranger...
I20260806 10:48:45.505398 16468 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 10:48:45.986115 16468 mini_ranger.cc:87] Created ranger Postgres database
I20260806 10:48:45.986279 16468 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-admin
2026-08-06 10:48:46,317 [I] DB FLAVOR :POSTGRES
2026-08-06 10:48:46,317 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:48:46,317 [I] Checking connection..
2026-08-06 10:48:46,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 10:48:46,566 [I] Checking connection passed.
2026-08-06 10:48:46,566 [I] --------- Verifying version history table ---------
2026-08-06 10:48:46,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:48:46,826 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:48:46,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:48:47,083 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:48:47,083 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 10:48:47,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 10:48:47.308 UTC [27593] WARNING: there is no transaction in progress
2026-08-06 10:48:47,315 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 10:48:47,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:48:47,563 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 10:48:47,563 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:48:47,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 10:48:47,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 10:48:48,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0b58.c.gcp-upstream.internal','N') ;"
2026-08-06 10:48:48,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:48:48,541 [I] Table x_portal_user does not exist in database ranger
2026-08-06 10:48:48,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:48:48,806 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 10:48:48,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:48:49,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:48:49,275 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 10:48:49,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 10:48:50.174 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.189 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.204 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.219 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.235 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.254 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.304 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.310 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.324 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.334 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.344 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.357 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.366 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.372 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.379 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:50.386 UTC [27703] WARNING: there is no transaction in progress
2026-08-06 10:48:51,006 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 10:48:51,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 10:48:51,260 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 10:48:51,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:48:51,520 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 10:48:51,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:48:51,762 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 10:48:51,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:48:52,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:48:52,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0b58.c.gcp-upstream.internal';"
2026-08-06 10:48:52,504 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 10:48:52,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 10:48:52,748 [I] DB_PATCHES have already been applied
I20260806 10:48:52.754990 16468 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLWW7rW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 10:48:52.755076 16468 mini_ranger.cc:193] Using host: 127.16.21.20
I20260806 10:48:52.778679 16468 mini_ranger.cc:227] Ranger admin URL: http://127.16.21.20:51563
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 10:48:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 10:48:53.918155 16468 mini_ranger.cc:160] Time spent starting Ranger: real 8.445s user 0.002s sys 0.064s
Aug 06, 2026 10:49:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 10:49:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 06, 2026 10:49:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 10:49:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 10:49:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 10:49:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 10:49:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 10:49:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 10:49:15.442963 16468 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/rangerkms_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/rangerkms_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.21.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.21.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 10:49:15 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013355, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 10:49:15.658309 16468 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 10:49:15.689167 16468 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 10:49:16.525637 16468 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 10:49:16.525822 16468 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms
I20260806 10:49:16.525977 16468 mini_ranger_kms.cc:227] Using postgres at 127.16.21.20:53865
2026-08-06 10:49:16,999 [I] DB FLAVOR :POSTGRES
2026-08-06 10:49:16,999 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:49:16,999 [I] Checking connection
2026-08-06 10:49:16,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 10:49:17,229 [I] connection success
2026-08-06 10:49:17,229 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 10:49:17,229 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 10:49:17,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 10:49:17,474 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 10:49:17,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:49:17,475 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 10:49:17,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:53865/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 10:49:17,729 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 10:49:18.184302 16468 mini_ranger_kms.cc:310] Created kms service
I20260806 10:49:18.407470 16468 mini_ranger_kms.cc:326] Created kudu user
I20260806 10:49:18.450527 16468 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 10:49:18 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013355, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:49:18.837347 16468 mini_ranger_kms.cc:384] Added ranger policy
I20260806 10:49:18.837812 16468 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 10:49:18.837893 16468 mini_ranger_kms.cc:240] Using host: 127.16.21.20
I20260806 10:49:18.867640 16468 mini_ranger_kms.cc:276] Ranger KMS PID: 27885
I20260806 10:49:18.867839 16468 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.21.20:55273
Aug 06, 2026 10:49:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:49:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:49:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 10:49:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 10:49:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 10:49:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 10:49:21 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.21.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:49:21.231387 16468 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.573s user 0.105s sys 0.416s
I20260806 10:49:21.231751 16468 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 10:49:21.231972 16468 mini_ranger_kms.cc:401] 127.16.21.20:55273/kms/v1/keys
Aug 06 10:49:21 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.21.20@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
Aug 06 10:49:21 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.21.20@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
Aug 06 10:49:23 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:49:24.067920 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38137
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:38137
--ranger_config_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:55273/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:49:24.411415 27936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:49:24.411662 27936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:49:24.411861 27936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:49:24.418999 27936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:49:24.419140 27936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:49:24.419248 27936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:49:24.419344 27936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:49:24.419435 27936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:49:24.429157 27936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.21.20:55273/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:38137
--ranger_config_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:49:24.430310 27936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:49:24.431562 27936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:49:24.442992 27941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:24.443135 27944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:24.443194 27942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:49:24.443509 27936 server_base.cc:1034] running on GCE node
I20260806 10:49:24.444416 27936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:49:24.445650 27936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:49:24.446903 27936 hybrid_clock.cc:648] HybridClock initialized: now 1786013364446855 us; error 38 us; skew 500 ppm
I20260806 10:49:24.447233 27936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:49:24 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013364, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:49:24.450237 27936 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:49:24.451546 27936 webserver.cc:460] Webserver started at http://127.16.21.62:44953/ using document root <none> and password file <none>
I20260806 10:49:24.452148 27936 fs_manager.cc:359] Metadata directory not provided
I20260806 10:49:24.452298 27936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:49:24.452600 27936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:49:24 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013364, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:49:24.502574 27936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "9ef75d4f34bd41c8a5735b960d8db48c"
format_stamp: "Formatted at 2026-08-06 10:49:24 on dist-test-slave-0b58"
server_key: "5c35ce18d39fe6ba68fd5801b7384840"
server_key_iv: "62e51a307d4df1c3ec2d571ec8feef6c"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:24.503371 27936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "9ef75d4f34bd41c8a5735b960d8db48c"
format_stamp: "Formatted at 2026-08-06 10:49:24 on dist-test-slave-0b58"
server_key: "5c35ce18d39fe6ba68fd5801b7384840"
server_key_iv: "62e51a307d4df1c3ec2d571ec8feef6c"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:24.508754 27936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:49:24.523983 27951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:24.525529 27936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:49:24.525723 27936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "9ef75d4f34bd41c8a5735b960d8db48c"
format_stamp: "Formatted at 2026-08-06 10:49:24 on dist-test-slave-0b58"
server_key: "5c35ce18d39fe6ba68fd5801b7384840"
server_key_iv: "62e51a307d4df1c3ec2d571ec8feef6c"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:24.525944 27936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:49:24.578989 27936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:49:24.582835 27936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:49:24.583155 27936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:49:24.601827 27936 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:38137
I20260806 10:49:24.601909 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:38137 every 8 connection(s)
I20260806 10:49:24.603163 27936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:49:24.607016 28003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:49:24.612648 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 27936
I20260806 10:49:24.612865 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:49:24.613492 16468 external_mini_cluster.cc:1428] Setting key 761fe432f9b5cc9042d7722b9d12626a
I20260806 10:49:24.617980 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c: Bootstrap starting.
I20260806 10:49:24.621495 28003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c: Neither blocks nor log segments found. Creating new log.
I20260806 10:49:24.622731 28003 log.cc:824] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:49:24.625571 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c: No bootstrap required, opened a new log
Aug 06 10:49:24 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:49:24.633889 28003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38137 } }
I20260806 10:49:24.634244 28003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:49:24.634373 28003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef75d4f34bd41c8a5735b960d8db48c, State: Initialized, Role: FOLLOWER
I20260806 10:49:24.635035 28003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38137 } }
I20260806 10:49:24.635316 28003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:49:24.635480 28003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:49:24.635663 28003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:49:24.635892 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:24.615514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.16.21.62:38137)
0806 10:49:24.616404 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:24.616416 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:24.616456 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:49:24.618051 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:24.618071 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:24.618473 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:24.618782 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:24.620421 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:24.621778 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:24.623615 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:24.624345 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:24.624546 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:24.628665 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:24.628673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:24.628689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:24.628732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:24.631829 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:24.632593 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:24.632599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:24.632604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:24.632752 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:24.633149 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:24.633153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:24.633155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:24.633639 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:24.633942 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:24.634415 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 10:49:24.634766 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":451,"threads_started":1}
I20260806 10:49:24.638662 28003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38137 } }
I20260806 10:49:24.639237 28003 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef75d4f34bd41c8a5735b960d8db48c; no voters:
I20260806 10:49:24.639739 28003 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:49:24.639891 28008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:49:24.641448 28008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [term 1 LEADER]: Becoming Leader. State: Replica: 9ef75d4f34bd41c8a5735b960d8db48c, State: Running, Role: LEADER
I20260806 10:49:24.641983 28008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38137 } }
I20260806 10:49:24.642257 28003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:49:24.644799 28009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38137 } } }
I20260806 10:49:24.644829 28010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef75d4f34bd41c8a5735b960d8db48c. Latest consensus state: current_term: 1 leader_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef75d4f34bd41c8a5735b960d8db48c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38137 } } }
I20260806 10:49:24.645360 28010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [sys.catalog]: This master's current role is: LEADER
I20260806 10:49:24.645360 28009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c [sys.catalog]: This master's current role is: LEADER
I20260806 10:49:24.647497 28014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:49:24.655036 28014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:49:24.655745 28003 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 10:49:24.673442 28014 catalog_manager.cc:1347] Generated new cluster ID: b6d2a497921b43c3a23d1893a19ddb82
I20260806 10:49:24.673578 28014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:49:24.698904 28014 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:49:24.700901 28014 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:49:24.710057 28014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c: Generated new TSK 0
I20260806 10:49:24.711066 28014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:49:25.411367 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:55273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38137
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:49:25 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 10:49:25.819260 28048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:49:25.819610 28048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:49:25.819820 28048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:49:25.831105 28048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:49:25.831255 28048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:49:25.831453 28048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:49:25.845432 28048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.21.20:55273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:49:25.846730 28048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:49:25.848073 28048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:49:25.864140 28059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:25.865453 28058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:25.865816 28061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:49:25.867730 28048 server_base.cc:1034] running on GCE node
I20260806 10:49:25.868538 28048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:49:25.869822 28048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:49:25.871227 28048 hybrid_clock.cc:648] HybridClock initialized: now 1786013365871157 us; error 47 us; skew 500 ppm
I20260806 10:49:25.871591 28048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:49:25 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:49:25.875552 28048 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:49:25.877431 28048 webserver.cc:460] Webserver started at http://127.16.21.1:38385/ using document root <none> and password file <none>
I20260806 10:49:25.878366 28048 fs_manager.cc:359] Metadata directory not provided
I20260806 10:49:25.878545 28048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:49:25.878968 28048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:49:25 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:49:25.907567 28048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a"
format_stamp: "Formatted at 2026-08-06 10:49:25 on dist-test-slave-0b58"
server_key: "8b1c432476cfbbcff5405fe7e3658e41"
server_key_iv: "7989ac80ddd9c2b85ac3f8d666a2b4c1"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:25.911211 28048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a"
format_stamp: "Formatted at 2026-08-06 10:49:25 on dist-test-slave-0b58"
server_key: "8b1c432476cfbbcff5405fe7e3658e41"
server_key_iv: "7989ac80ddd9c2b85ac3f8d666a2b4c1"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:25.918668 28048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:49:25.941288 28068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:25.942907 28048 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s
I20260806 10:49:25.943147 28048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a"
format_stamp: "Formatted at 2026-08-06 10:49:25 on dist-test-slave-0b58"
server_key: "8b1c432476cfbbcff5405fe7e3658e41"
server_key_iv: "7989ac80ddd9c2b85ac3f8d666a2b4c1"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:25.943393 28048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:49:25.985219 28048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:49:25.989346 28048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:49:25.989665 28048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:49:25.990623 28048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:49:25.992117 28048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:49:25.992218 28048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:25.992538 28048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:49:25.992623 28048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:26.061218 28048 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:34121
I20260806 10:49:26.061266 28180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:34121 every 8 connection(s)
I20260806 10:49:26.063058 28048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:49:26.064950 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 28048
I20260806 10:49:26.065155 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:49:26.066005 16468 external_mini_cluster.cc:1428] Setting key a136690e5ce591e5df6a75cdc94fa46b
Aug 06 10:49:26 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:49:26.111753 28183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:26.078625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38137 (local address 127.16.21.1:33465)
0806 10:49:26.086871 (+ 8246us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:26.086926 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:26.090131 (+ 3205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:26.092071 (+ 1940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:26.092085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:26.092786 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:49:26.093756 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:26.093785 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.095390 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.095396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:26.097584 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:26.097600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.098914 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.098922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:26.099223 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:26.100030 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:49:26.100062 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:49:26.102903 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:49:26.107043 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:49:26.107052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:49:26.107069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:49:26.107653 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:49:26.108111 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:49:26.108118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:49:26.108121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:49:26.108398 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:49:26.109132 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:49:26.109140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:49:26.109532 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 10:49:26.109805 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 10:49:26.110332 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7909,"thread_start_us":7672,"threads_started":1}
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
I20260806 10:49:26.113144 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:26.079297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:33465 (local address 127.16.21.62:38137)
0806 10:49:26.088520 (+ 9223us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:26.088527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:26.088550 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:49:26.090465 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:26.090471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:26.090580 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:26.090710 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:26.094005 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.095171 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.097775 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.098432 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.098574 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:26.103165 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:26.103172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:26.103178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:26.103245 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:26.106874 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:26.107800 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:26.107806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:26.107809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:26.107967 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:26.108543 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:26.108548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:26.108551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:26.108894 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:26.109060 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:26.112533 (+ 3473us) server_negotiation.cc:299] Negotiation successful
0806 10:49:26.112772 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9098,"thread_start_us":8995,"threads_started":1}
I20260806 10:49:26.114876 28181 heartbeater.cc:347] Connected to a master server at 127.16.21.62:38137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
I20260806 10:49:26.115286 28181 heartbeater.cc:464] Registering TS with master...
I20260806 10:49:26.116218 28181 heartbeater.cc:511] Master 127.16.21.62:38137 requested a full tablet report, sending...
I20260806 10:49:26.118903 27968 ts_manager.cc:194] Registered new tserver with Master: da6e9d3d0ddb43b1a2e4c09e4ab8352a (127.16.21.1:34121)
I20260806 10:49:26.120929 27968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:33465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:49:26.152436 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:55273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38137
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:49:26 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
Aug 06 10:49:26 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
W20260806 10:49:26.624357 28187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:49:26.624667 28187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:49:26.624850 28187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:49:26.633770 28187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:49:26.633909 28187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:49:26.634091 28187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:49:26.644424 28187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.21.20:55273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:49:26.646143 28187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:49:26.647773 28187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:49:26.675292 28194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:26.676625 28195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:26.684195 28197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:49:26.685384 28187 server_base.cc:1034] running on GCE node
I20260806 10:49:26.687606 28187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:49:26.696557 28187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:49:26.699922 28187 hybrid_clock.cc:648] HybridClock initialized: now 1786013366699805 us; error 108 us; skew 500 ppm
I20260806 10:49:26.700341 28187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:49:26 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013366, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:49:26.704522 28187 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:49:26.728354 28187 webserver.cc:460] Webserver started at http://127.16.21.2:45279/ using document root <none> and password file <none>
I20260806 10:49:26.729494 28187 fs_manager.cc:359] Metadata directory not provided
I20260806 10:49:26.729714 28187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:49:26.730172 28187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:49:26 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013366, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:49:26.764963 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "a2e4218a215a474ba81e8782d11b755f"
format_stamp: "Formatted at 2026-08-06 10:49:26 on dist-test-slave-0b58"
server_key: "3bd9798cbd68361749e2725d27a96fe0"
server_key_iv: "806d1becb5daf7afb3221d71934145a7"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:26.773123 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "a2e4218a215a474ba81e8782d11b755f"
format_stamp: "Formatted at 2026-08-06 10:49:26 on dist-test-slave-0b58"
server_key: "3bd9798cbd68361749e2725d27a96fe0"
server_key_iv: "806d1becb5daf7afb3221d71934145a7"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:26.790371 28187 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.008s sys 0.008s
I20260806 10:49:26.822214 28204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:26.825274 28187 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.009s sys 0.004s
I20260806 10:49:26.825471 28187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "a2e4218a215a474ba81e8782d11b755f"
format_stamp: "Formatted at 2026-08-06 10:49:26 on dist-test-slave-0b58"
server_key: "3bd9798cbd68361749e2725d27a96fe0"
server_key_iv: "806d1becb5daf7afb3221d71934145a7"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:26.825744 28187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:49:26.874045 28187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:49:26.879379 28187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:49:26.879806 28187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:49:26.881107 28187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:49:26.882984 28187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:49:26.883121 28187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:26.883293 28187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:49:26.883375 28187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:26.944638 28187 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:46303
I20260806 10:49:26.944674 28325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:46303 every 8 connection(s)
I20260806 10:49:26.945887 28187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:49:26.954030 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 28187
I20260806 10:49:26.954216 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:49:26.954663 16468 external_mini_cluster.cc:1428] Setting key 11f353a697421c3d63c858770d8345ca
Aug 06 10:49:26 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013366, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:49:26.965747 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:26.948422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:49583 (local address 127.16.21.62:38137)
0806 10:49:26.948881 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:26.948885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:26.950054 (+ 1169us) server_negotiation.cc:407] Connection header received
0806 10:49:26.951911 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:26.951917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:26.952024 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:26.952143 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:26.953542 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.954378 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.956144 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.956699 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.956844 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:26.960300 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:26.960312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:26.960317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:26.960347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:26.962809 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:26.963454 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:26.963461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:26.963464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:26.963585 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:26.964027 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:26.964032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:26.964035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:26.964361 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:26.964511 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:26.965120 (+ 609us) server_negotiation.cc:299] Negotiation successful
0806 10:49:26.965378 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":210,"threads_started":1}
I20260806 10:49:26.966588 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:26.949104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38137 (local address 127.16.21.2:49583)
0806 10:49:26.949908 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:26.949945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:26.951656 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:26.952413 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:26.952421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:26.952849 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:49:26.953378 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:26.953395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.954539 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.954543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:26.955993 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:26.956005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:26.956934 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:26.956941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:26.957156 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:26.957817 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:49:26.957837 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:49:26.960057 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:49:26.962944 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:49:26.962951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:49:26.962965 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:49:26.963339 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:49:26.963721 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:49:26.963728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:49:26.963732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:49:26.963916 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:49:26.964486 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:49:26.964493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:49:26.964794 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 10:49:26.965031 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 10:49:26.965509 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260806 10:49:26.967867 28326 heartbeater.cc:347] Connected to a master server at 127.16.21.62:38137
I20260806 10:49:26.968135 28326 heartbeater.cc:464] Registering TS with master...
I20260806 10:49:26.968777 28326 heartbeater.cc:511] Master 127.16.21.62:38137 requested a full tablet report, sending...
I20260806 10:49:26.970327 27968 ts_manager.cc:194] Registered new tserver with Master: a2e4218a215a474ba81e8782d11b755f (127.16.21.2:46303)
I20260806 10:49:26.971122 27968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:49583
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:49:27.018724 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:55273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38137
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:49:27.124498 28181 heartbeater.cc:503] Master 127.16.21.62:38137 was elected leader, sending a full tablet report...
W20260806 10:49:27.348091 28332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:49:27.348441 28332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:49:27.348604 28332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:49:27.355783 28332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:49:27.355927 28332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:49:27.356091 28332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:49:27.365047 28332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.21.20:55273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:49:27.366240 28332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:49:27.367451 28332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:49:27.380138 28338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:27.380155 28337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:49:27.380856 28340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:49:27.380955 28332 server_base.cc:1034] running on GCE node
I20260806 10:49:27.381498 28332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:49:27.382334 28332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:49:27.383548 28332 hybrid_clock.cc:648] HybridClock initialized: now 1786013367383508 us; error 50 us; skew 500 ppm
I20260806 10:49:27.383912 28332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:49:27 dist-test-slave-0b58 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:49:27.391244 28332 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:49:27.392588 28332 webserver.cc:460] Webserver started at http://127.16.21.3:38643/ using document root <none> and password file <none>
I20260806 10:49:27.393265 28332 fs_manager.cc:359] Metadata directory not provided
I20260806 10:49:27.393421 28332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:49:27.393747 28332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:49:27 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:49:27.410629 28332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f"
format_stamp: "Formatted at 2026-08-06 10:49:27 on dist-test-slave-0b58"
server_key: "b66840d0c1ea05d13f104d49e2c2ec1d"
server_key_iv: "4814318be296d283e8562c6feaa22f24"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:27.411646 28332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f"
format_stamp: "Formatted at 2026-08-06 10:49:27 on dist-test-slave-0b58"
server_key: "b66840d0c1ea05d13f104d49e2c2ec1d"
server_key_iv: "4814318be296d283e8562c6feaa22f24"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:27.417822 28332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260806 10:49:27.435410 28347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:27.436934 28332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:49:27.437119 28332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f"
format_stamp: "Formatted at 2026-08-06 10:49:27 on dist-test-slave-0b58"
server_key: "b66840d0c1ea05d13f104d49e2c2ec1d"
server_key_iv: "4814318be296d283e8562c6feaa22f24"
server_key_version: "kuduclusterkey@0"
I20260806 10:49:27.437332 28332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:49:27.455857 28332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:49:27.459892 28332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:49:27.460151 28332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:49:27.461094 28332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:49:27.462407 28332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:49:27.462507 28332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:27.462651 28332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:49:27.462721 28332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:49:27.496558 28332 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:45721
I20260806 10:49:27.496608 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:45721 every 8 connection(s)
I20260806 10:49:27.497771 28332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:49:27.506486 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 28332
I20260806 10:49:27.506657 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:49:27.507103 16468 external_mini_cluster.cc:1428] Setting key 9c426afaebc02ffb153a6763c8e8c637
Aug 06 10:49:27 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:49:27.516389 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.500163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:44599 (local address 127.16.21.62:38137)
0806 10:49:27.500715 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.500719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.501524 (+ 805us) server_negotiation.cc:407] Connection header received
0806 10:49:27.503267 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.503274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.503360 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.503476 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:27.504893 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.505627 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.507307 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.507746 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.507888 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.511078 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:27.511083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:27.511086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:27.511115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:27.513489 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:27.514160 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:27.514165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:27.514167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:27.514288 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:27.514719 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:27.514724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:27.514727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:27.514990 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:27.515157 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.515791 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.516000 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":284,"threads_started":1}
I20260806 10:49:27.517297 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.500697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38137 (local address 127.16.21.3:44599)
0806 10:49:27.501368 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.501404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.503008 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.503718 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.503726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.504118 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:49:27.504725 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.504742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.505791 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.505795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.507129 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.507162 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.507980 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.507988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.508213 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.508913 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:49:27.508934 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:49:27.510838 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:49:27.513615 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:49:27.513622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:49:27.513634 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:49:27.514015 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:49:27.514395 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:49:27.514400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:49:27.514403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:49:27.514601 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:49:27.515124 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:49:27.515131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:49:27.515445 (+ 314us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.515695 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.516163 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260806 10:49:27.518488 28460 heartbeater.cc:347] Connected to a master server at 127.16.21.62:38137
I20260806 10:49:27.518723 28460 heartbeater.cc:464] Registering TS with master...
I20260806 10:49:27.519335 28460 heartbeater.cc:511] Master 127.16.21.62:38137 requested a full tablet report, sending...
I20260806 10:49:27.520730 27968 ts_manager.cc:194] Registered new tserver with Master: ebe5e7b0bc4c48d4911c7ff82eb08b1f (127.16.21.3:45721)
I20260806 10:49:27.521751 27968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:44599
I20260806 10:49:27.522398 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:49:27.540014 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.526840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.16.21.62:38137)
0806 10:49:27.526979 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.526983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.527477 (+ 494us) server_negotiation.cc:407] Connection header received
0806 10:49:27.527657 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.527669 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.527735 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.527837 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:27.529412 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.530083 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.531801 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.532367 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.532494 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.534965 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:27.534975 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:27.534979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:27.535014 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:27.537375 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:27.538065 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:27.538072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:27.538075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:27.538224 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:27.538606 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:27.538612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:27.538618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:27.538898 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:27.539151 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.539481 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.539694 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:49:27.543951 27968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:49:27.551182 27968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:49:27.584904 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.562865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46303 (local address 127.0.0.1:60378)
0806 10:49:27.575568 (+ 12703us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.575609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.575793 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.577073 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.577085 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.577114 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.577587 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.577602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.579455 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.579462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.581920 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.581933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.583761 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.583768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.584004 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.584055 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.584185 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.584395 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12394,"mutex_wait_us":114,"thread_start_us":12843,"threads_started":1}
I20260806 10:49:27.585649 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.563036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.16.21.2:46303)
0806 10:49:27.575672 (+ 12636us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.575681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.575711 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:49:27.576246 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.576283 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.576565 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.576773 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.577777 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.579317 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.582074 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.583539 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.583657 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.583846 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.584245 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.585263 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12455,"thread_start_us":12909,"threads_started":1}
I20260806 10:49:27.585875 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.576439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:45721 (local address 127.0.0.1:42064)
0806 10:49:27.577096 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.577118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.577290 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.578541 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.578546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.578571 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.578957 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.578970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.580823 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.580829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.583183 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.583198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.584944 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.584950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.585239 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.585270 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.585337 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.585467 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":54,"thread_start_us":1353,"threads_started":1}
I20260806 10:49:27.586515 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.576475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.16.21.3:45721)
0806 10:49:27.578099 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.578105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.578129 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:49:27.578206 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.578210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.578350 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.578561 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.579133 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.580660 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.583343 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.584684 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.585567 (+ 883us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.585807 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.585872 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.586023 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1348,"threads_started":1}
I20260806 10:49:27.589466 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.563623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.16.21.1:34121)
0806 10:49:27.576553 (+ 12930us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.576563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.577050 (+ 487us) server_negotiation.cc:407] Connection header received
0806 10:49:27.577149 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.577160 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.577383 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.577582 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.579482 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.580662 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.585255 (+ 4593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.587844 (+ 2589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.587942 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.588135 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.588970 (+ 835us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.589126 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12754,"thread_start_us":13741,"threads_started":1}
I20260806 10:49:27.589694 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.563007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34121 (local address 127.0.0.1:59156)
0806 10:49:27.576781 (+ 13774us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.576799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.576925 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.578821 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.578826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.578852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.579275 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.579287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.580891 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.580896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.584895 (+ 3999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.584906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.588372 (+ 3466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.588378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.588537 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.588569 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.588660 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.588788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13611,"mutex_wait_us":44,"thread_start_us":12149,"threads_started":1}
I20260806 10:49:27.589665 28259 tablet_service.cc:1467] Processing CreateTablet for tablet bbb58181a7c6436fbb749ef4d3daad1f (DEFAULT_TABLE table=test-table [id=23f15950f96e4148ad57ce50d1dc1c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:49:27.589802 28395 tablet_service.cc:1467] Processing CreateTablet for tablet bbb58181a7c6436fbb749ef4d3daad1f (DEFAULT_TABLE table=test-table [id=23f15950f96e4148ad57ce50d1dc1c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:49:27.591147 28395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb58181a7c6436fbb749ef4d3daad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:49:27.591178 28259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb58181a7c6436fbb749ef4d3daad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:49:27.593132 28116 tablet_service.cc:1467] Processing CreateTablet for tablet bbb58181a7c6436fbb749ef4d3daad1f (DEFAULT_TABLE table=test-table [id=23f15950f96e4148ad57ce50d1dc1c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:49:27.594801 28116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb58181a7c6436fbb749ef4d3daad1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:49:27.608207 28480 tablet_bootstrap.cc:492] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Bootstrap starting.
I20260806 10:49:27.611153 28481 tablet_bootstrap.cc:492] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Bootstrap starting.
I20260806 10:49:27.613430 28480 tablet_bootstrap.cc:654] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Neither blocks nor log segments found. Creating new log.
I20260806 10:49:27.614671 28480 log.cc:824] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:49:27.616899 28481 tablet_bootstrap.cc:654] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Neither blocks nor log segments found. Creating new log.
I20260806 10:49:27.618541 28481 log.cc:824] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:49:27.619771 28482 tablet_bootstrap.cc:492] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Bootstrap starting.
I20260806 10:49:27.622843 28482 tablet_bootstrap.cc:654] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Neither blocks nor log segments found. Creating new log.
I20260806 10:49:27.624078 28482 log.cc:824] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:49:27.624511 28481 tablet_bootstrap.cc:492] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: No bootstrap required, opened a new log
I20260806 10:49:27.624809 28481 ts_tablet_manager.cc:1397] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260806 10:49:27.627983 28480 tablet_bootstrap.cc:492] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: No bootstrap required, opened a new log
I20260806 10:49:27.628415 28480 ts_tablet_manager.cc:1397] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.008s
I20260806 10:49:27.631840 28482 tablet_bootstrap.cc:492] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: No bootstrap required, opened a new log
I20260806 10:49:27.632128 28482 ts_tablet_manager.cc:1397] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260806 10:49:27.633749 28481 raft_consensus.cc:348] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.634114 28481 raft_consensus.cc:374] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:49:27.634265 28481 raft_consensus.cc:729] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e4218a215a474ba81e8782d11b755f, State: Initialized, Role: FOLLOWER
I20260806 10:49:27.634857 28481 consensus_queue.cc:260] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.635968 28326 heartbeater.cc:503] Master 127.16.21.62:38137 was elected leader, sending a full tablet report...
I20260806 10:49:27.636615 28481 ts_tablet_manager.cc:1428] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260806 10:49:27.639551 28480 raft_consensus.cc:348] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.639966 28480 raft_consensus.cc:374] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:49:27.640057 28480 raft_consensus.cc:729] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe5e7b0bc4c48d4911c7ff82eb08b1f, State: Initialized, Role: FOLLOWER
I20260806 10:49:27.640702 28480 consensus_queue.cc:260] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.641892 28460 heartbeater.cc:503] Master 127.16.21.62:38137 was elected leader, sending a full tablet report...
I20260806 10:49:27.642501 28480 ts_tablet_manager.cc:1428] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 10:49:27.642886 28482 raft_consensus.cc:348] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.643304 28482 raft_consensus.cc:374] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:49:27.643425 28482 raft_consensus.cc:729] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6e9d3d0ddb43b1a2e4c09e4ab8352a, State: Initialized, Role: FOLLOWER
I20260806 10:49:27.644100 28482 consensus_queue.cc:260] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.645795 28482 ts_tablet_manager.cc:1428] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260806 10:49:27.697867 28327 tablet.cc:2369] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:49:27.749428 28461 tablet.cc:2369] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:49:27.822049 28182 tablet.cc:2369] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:49:27.846570 28486 raft_consensus.cc:482] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:49:27.846884 28486 raft_consensus.cc:504] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.848932 28486 leader_election.cc:288] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da6e9d3d0ddb43b1a2e4c09e4ab8352a (127.16.21.1:34121), ebe5e7b0bc4c48d4911c7ff82eb08b1f (127.16.21.3:45721)
I20260806 10:49:27.858222 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.849597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:56975 (local address 127.16.21.3:45721)
0806 10:49:27.849839 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.849847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.850196 (+ 349us) server_negotiation.cc:407] Connection header received
0806 10:49:27.850452 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.850457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.850562 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.850661 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.851460 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.852606 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.855688 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.857035 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.857169 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.857284 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.857831 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.857970 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:49:27.858253 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.849268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:57429 (local address 127.16.21.1:34121)
0806 10:49:27.849516 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.849523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.850457 (+ 934us) server_negotiation.cc:407] Connection header received
0806 10:49:27.850633 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.850641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.850714 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.850801 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.851461 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.852606 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.855699 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.857020 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.857170 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.857284 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.857831 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.857970 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:49:27.858515 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.849304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34121 (local address 127.16.21.2:57429)
0806 10:49:27.850377 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.850391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.850496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.850834 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.850841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.850879 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.851316 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.851329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.852751 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.852754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.855488 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.855504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.857345 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.857351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.857585 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.857680 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.857776 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.857987 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":279,"thread_start_us":758,"threads_started":1}
I20260806 10:49:27.858541 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.849273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:45721 (local address 127.16.21.2:56975)
0806 10:49:27.850079 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.850094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.850295 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.850688 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.850692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.850711 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.851268 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.851283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.852749 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.852754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.855493 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.855504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.857350 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.857360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.857569 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.857674 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.857776 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.857987 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":248,"thread_start_us":480,"threads_started":1}
I20260806 10:49:27.859827 28136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb58181a7c6436fbb749ef4d3daad1f" candidate_uuid: "a2e4218a215a474ba81e8782d11b755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" is_pre_election: true
I20260806 10:49:27.860131 28488 raft_consensus.cc:482] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:49:27.860159 28415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb58181a7c6436fbb749ef4d3daad1f" candidate_uuid: "a2e4218a215a474ba81e8782d11b755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" is_pre_election: true
I20260806 10:49:27.860461 28488 raft_consensus.cc:504] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.860739 28415 raft_consensus.cc:2454] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e4218a215a474ba81e8782d11b755f in term 0.
I20260806 10:49:27.861213 28136 raft_consensus.cc:2454] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e4218a215a474ba81e8782d11b755f in term 0.
I20260806 10:49:27.861439 28205 leader_election.cc:302] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e4218a215a474ba81e8782d11b755f, ebe5e7b0bc4c48d4911c7ff82eb08b1f; no voters:
I20260806 10:49:27.861912 28486 raft_consensus.cc:2781] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:49:27.862073 28486 raft_consensus.cc:482] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:49:27.862185 28486 raft_consensus.cc:3037] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:49:27.862421 28488 leader_election.cc:288] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e4218a215a474ba81e8782d11b755f (127.16.21.2:46303), ebe5e7b0bc4c48d4911c7ff82eb08b1f (127.16.21.3:45721)
I20260806 10:49:27.865267 28486 raft_consensus.cc:504] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.866078 28486 leader_election.cc:288] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [CANDIDATE]: Term 1 election: Requested vote from peers da6e9d3d0ddb43b1a2e4c09e4ab8352a (127.16.21.1:34121), ebe5e7b0bc4c48d4911c7ff82eb08b1f (127.16.21.3:45721)
I20260806 10:49:27.866581 28415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb58181a7c6436fbb749ef4d3daad1f" candidate_uuid: "a2e4218a215a474ba81e8782d11b755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f"
I20260806 10:49:27.866804 28415 raft_consensus.cc:3037] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:49:27.867416 28136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb58181a7c6436fbb749ef4d3daad1f" candidate_uuid: "a2e4218a215a474ba81e8782d11b755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a"
I20260806 10:49:27.867621 28136 raft_consensus.cc:3037] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:49:27.869654 28415 raft_consensus.cc:2454] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e4218a215a474ba81e8782d11b755f in term 1.
I20260806 10:49:27.870405 28136 raft_consensus.cc:2454] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e4218a215a474ba81e8782d11b755f in term 1.
I20260806 10:49:27.870628 28205 leader_election.cc:302] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e4218a215a474ba81e8782d11b755f, ebe5e7b0bc4c48d4911c7ff82eb08b1f; no voters:
I20260806 10:49:27.870963 28486 raft_consensus.cc:2781] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:49:27.871217 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.862779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:47259 (local address 127.16.21.2:46303)
0806 10:49:27.862986 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.862992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.863420 (+ 428us) server_negotiation.cc:407] Connection header received
0806 10:49:27.863740 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.863748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.863863 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.863940 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.864884 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.865984 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.868886 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.870097 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.870274 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.870370 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.870738 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.870860 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:49:27.871667 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.862559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:46303 (local address 127.16.21.1:47259)
0806 10:49:27.863333 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.863350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.863531 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.864168 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.864176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.864205 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.864749 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.864759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.866486 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.866491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.868721 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.868740 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.870341 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.870347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.870538 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.870573 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.870999 (+ 426us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.871213 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":51,"thread_start_us":449,"threads_started":1}
I20260806 10:49:27.871878 28486 raft_consensus.cc:686] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 1 LEADER]: Becoming Leader. State: Replica: a2e4218a215a474ba81e8782d11b755f, State: Running, Role: LEADER
I20260806 10:49:27.872439 28486 consensus_queue.cc:237] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } }
I20260806 10:49:27.873016 28279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb58181a7c6436fbb749ef4d3daad1f" candidate_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e4218a215a474ba81e8782d11b755f" is_pre_election: true
I20260806 10:49:27.875165 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.863506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:37223 (local address 127.16.21.3:45721)
0806 10:49:27.863657 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.863662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.864938 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 10:49:27.865161 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.865166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.865253 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.865346 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:49:27.865933 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.867113 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.872940 (+ 5827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.873854 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.873970 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.874082 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.874784 (+ 702us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.874921 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:49:27.875233 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.863413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:45721 (local address 127.16.21.1:37223)
0806 10:49:27.864843 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0806 10:49:27.864866 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:49:27.864997 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:49:27.865388 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:49:27.865393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:49:27.865420 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:49:27.865773 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.865784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.870669 (+ 4885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.870674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.872782 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:49:27.872796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.874407 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.874411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:49:27.874579 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.874622 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 10:49:27.874741 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 10:49:27.874867 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":4,"thread_start_us":1073,"threads_started":1}
I20260806 10:49:27.876660 28415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb58181a7c6436fbb749ef4d3daad1f" candidate_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" is_pre_election: true
I20260806 10:49:27.876950 28415 raft_consensus.cc:2379] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate da6e9d3d0ddb43b1a2e4c09e4ab8352a in current term 1: Already voted for candidate a2e4218a215a474ba81e8782d11b755f in this term.
I20260806 10:49:27.877530 28069 leader_election.cc:302] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6e9d3d0ddb43b1a2e4c09e4ab8352a; no voters: a2e4218a215a474ba81e8782d11b755f, ebe5e7b0bc4c48d4911c7ff82eb08b1f
I20260806 10:49:27.877929 28488 raft_consensus.cc:2726] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 10:49:27.877521 27968 catalog_manager.cc:5495] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e4218a215a474ba81e8782d11b755f (127.16.21.2). New cstate: current_term: 1 leader_uuid: "a2e4218a215a474ba81e8782d11b755f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4218a215a474ba81e8782d11b755f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 } health_report { overall_health: UNKNOWN } } }
I20260806 10:49:27.955248 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:27.949874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.16.21.2:46303)
0806 10:49:27.950056 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:27.950060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:27.950076 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:49:27.950147 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:27.950151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:27.950256 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:27.950413 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:49:27.950813 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.951575 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.953067 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:27.953528 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:27.953685 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:27.954162 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:49:27.954619 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:49:27.954797 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:27.954850 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 10:49:27.954929 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:49:27.963682 28415 raft_consensus.cc:1261] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 1 FOLLOWER]: Refusing update from remote peer a2e4218a215a474ba81e8782d11b755f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:49:27.963825 28136 raft_consensus.cc:1261] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 1 FOLLOWER]: Refusing update from remote peer a2e4218a215a474ba81e8782d11b755f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:49:27.964636 28493 consensus_queue.cc:1035] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe5e7b0bc4c48d4911c7ff82eb08b1f" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:49:27.964967 28486 consensus_queue.cc:1035] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6e9d3d0ddb43b1a2e4c09e4ab8352a" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:49:27.980861 28497 mvcc.cc:205] Tried to move back new op lower bound from 7315510755170193408 to 7315510754810273792. Current Snapshot: MvccSnapshot[applied={T|T < 7315510755170193408}]
I20260806 10:49:27.990512 27968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:49:27.990746 27968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:49:27.994222 27968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef75d4f34bd41c8a5735b960d8db48c: Sending DeleteTablet for 3 replicas of tablet bbb58181a7c6436fbb749ef4d3daad1f
I20260806 10:49:27.995301 28259 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb58181a7c6436fbb749ef4d3daad1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:49:27 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:60378
I20260806 10:49:27.995402 28116 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb58181a7c6436fbb749ef4d3daad1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:49:27 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:59156
I20260806 10:49:27.996315 28506 tablet_replica.cc:330] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: stopping tablet replica
I20260806 10:49:27.996430 28395 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb58181a7c6436fbb749ef4d3daad1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:49:27 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:42064
I20260806 10:49:27.996776 28506 raft_consensus.cc:2229] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:49:27.997895 28506 raft_consensus.cc:2258] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:49:27.999034 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 28048
I20260806 10:49:27.999949 28506 ts_tablet_manager.cc:1905] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:49:28.008922 28506 ts_tablet_manager.cc:1918] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:49:28.009150 28506 log.cc:1196] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/bbb58181a7c6436fbb749ef4d3daad1f
I20260806 10:49:28.009846 28506 ts_tablet_manager.cc:1939] T bbb58181a7c6436fbb749ef4d3daad1f P a2e4218a215a474ba81e8782d11b755f: Deleting consensus metadata
I20260806 10:49:28.011827 27952 catalog_manager.cc:4841] TS a2e4218a215a474ba81e8782d11b755f (127.16.21.2:46303): tablet bbb58181a7c6436fbb749ef4d3daad1f (table test-table [id=23f15950f96e4148ad57ce50d1dc1c29]) successfully deleted
I20260806 10:49:28.017531 28508 tablet_replica.cc:330] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: stopping tablet replica
I20260806 10:49:28.017800 28507 tablet_replica.cc:330] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: stopping tablet replica
I20260806 10:49:28.018039 28508 raft_consensus.cc:2229] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:49:28.018236 28507 raft_consensus.cc:2229] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:49:28.018539 28508 raft_consensus.cc:2258] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:49:28.018663 28507 raft_consensus.cc:2258] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:49:28.020931 28507 ts_tablet_manager.cc:1905] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:49:28.021504 28508 ts_tablet_manager.cc:1905] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 10:49:28 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:49:28.028007 28507 ts_tablet_manager.cc:1918] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:49:28.028304 28507 log.cc:1196] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/bbb58181a7c6436fbb749ef4d3daad1f
I20260806 10:49:28.028705 28508 ts_tablet_manager.cc:1918] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:49:28.028906 28507 ts_tablet_manager.cc:1939] T bbb58181a7c6436fbb749ef4d3daad1f P da6e9d3d0ddb43b1a2e4c09e4ab8352a: Deleting consensus metadata
I20260806 10:49:28.028919 28508 log.cc:1196] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/bbb58181a7c6436fbb749ef4d3daad1f
I20260806 10:49:28.029477 28508 ts_tablet_manager.cc:1939] T bbb58181a7c6436fbb749ef4d3daad1f P ebe5e7b0bc4c48d4911c7ff82eb08b1f: Deleting consensus metadata
I20260806 10:49:28.031033 27953 catalog_manager.cc:4841] TS da6e9d3d0ddb43b1a2e4c09e4ab8352a (127.16.21.1:34121): tablet bbb58181a7c6436fbb749ef4d3daad1f (table test-table [id=23f15950f96e4148ad57ce50d1dc1c29]) successfully deleted
I20260806 10:49:28.031405 27952 catalog_manager.cc:4841] TS ebe5e7b0bc4c48d4911c7ff82eb08b1f (127.16.21.3:45721): tablet bbb58181a7c6436fbb749ef4d3daad1f (table test-table [id=23f15950f96e4148ad57ce50d1dc1c29]) successfully deleted
I20260806 10:49:28.035465 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:28.001019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.16.21.1:34121)
0806 10:49:28.001213 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:28.001222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:28.016863 (+ 15641us) server_negotiation.cc:407] Connection header received
0806 10:49:28.017087 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:28.017096 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:28.017235 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:28.017383 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:28.019620 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:28.020440 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:28.022422 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:28.023115 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:28.023214 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:28.028823 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:28.028840 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:28.028858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:28.028916 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:28.032651 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:28.033530 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:28.033537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:28.033539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:28.033656 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:28.034076 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:28.034079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:28.034081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:28.034425 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:28.034647 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:28.035013 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 10:49:28.035173 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 10:49:28.036321 28176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:49:28.128822 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 28048
I20260806 10:49:28.150858 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 28187
Aug 06 10:49:28 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:49:28.168193 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:28.151654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.16.21.2:46303)
0806 10:49:28.151834 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:28.151839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:28.151868 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:49:28.152028 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:28.152034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:28.152139 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:28.152280 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:28.153596 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:28.154697 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:28.155989 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:28.156676 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:28.156814 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:28.161121 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:28.161139 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:28.161159 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:28.161201 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:28.165331 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:28.165968 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:28.165977 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:28.165981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:28.166194 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:28.166534 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:28.166540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:28.166543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:28.167035 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:28.167412 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:28.167578 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 10:49:28.167799 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:49:28.168998 28321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:49:28.287192 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 28187
I20260806 10:49:28.306578 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 28332
Aug 06 10:49:28 dist-test-slave-0b58 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:49:28.321164 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:49:28.307362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.16.21.3:45721)
0806 10:49:28.307527 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:49:28.307531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:49:28.307565 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:49:28.307733 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:49:28.307738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:49:28.307805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:49:28.307886 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:49:28.309249 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:28.309937 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:28.311120 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:49:28.311546 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:49:28.311636 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:49:28.315995 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:49:28.316008 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:49:28.316025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:49:28.316077 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:49:28.318697 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:28.319294 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:28.319302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:28.319304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:28.319445 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:49:28.319760 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:49:28.319763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:49:28.319765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:49:28.320116 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:49:28.320440 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 10:49:28.320735 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 10:49:28.320914 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:49:28.321781 28455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:49:28.410090 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 28332
I20260806 10:49:28.428529 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 27936
I20260806 10:49:28.430042 27998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:49:28.541992 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 27936
I20260806 10:49:28.568887 16468 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 10:49:28.913336 16468 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 10:49:28.913650 16468 mini_ranger.cc:66] Stopping Ranger...
I20260806 10:49:29.258172 16468 mini_ranger.cc:68] Stopped Ranger
2026-08-06 10:49:29.258 UTC [27517] LOG: received smart shutdown request
2026-08-06 10:49:29.261 UTC [27517] LOG: background worker "logical replication launcher" (PID 27526) exited with exit code 1
2026-08-06 10:49:29.261 UTC [27521] LOG: shutting down
2026-08-06 10:49:29.414 UTC [27517] LOG: database system is shut down
2026-08-06T10:49:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49109 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:49:29 dist-test-slave-0b58 krb5kdc[28522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:49:29 dist-test-slave-0b58 krb5kdc[28522](info): set up 2 sockets
Aug 06 10:49:29 dist-test-slave-0b58 krb5kdc[28522](info): commencing operation
krb5kdc: starting...
W20260806 10:49:30.672993 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:49:30 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:49:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/rangeradmin_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/rangeradmin_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/rangerlookup_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/rangerlookup_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
I20260806 10:49:31.063097 16468 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLWW7rW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/postgres -l logfile start
2026-08-06 10:49:33.495 UTC [28552] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 10:49:33.496 UTC [28552] LOG: listening on IPv4 address "127.16.21.20", port 39547
2026-08-06 10:49:33.499 UTC [28552] LOG: listening on Unix socket "/tmp/.s.PGSQL.39547"
2026-08-06 10:49:33.523 UTC [28554] LOG: database system was shut down at 2026-08-06 10:49:32 UTC
2026-08-06 10:49:33.527 UTC [28552] LOG: database system is ready to accept connections
I20260806 10:49:34.527667 16468 mini_postgres.cc:96] Postgres bound to 39547
2026-08-06 10:49:34.557 UTC [28563] FATAL: database "slave" does not exist
127.16.21.20:39547 - accepting connections
I20260806 10:49:34.558030 16468 mini_ranger.cc:161] Starting Ranger...
I20260806 10:49:34.587631 16468 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 10:49:35.053805 16468 mini_ranger.cc:87] Created ranger Postgres database
I20260806 10:49:35.053951 16468 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-admin
2026-08-06 10:49:35,344 [I] DB FLAVOR :POSTGRES
2026-08-06 10:49:35,345 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:49:35,345 [I] Checking connection..
2026-08-06 10:49:35,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 10:49:35,605 [I] Checking connection passed.
2026-08-06 10:49:35,605 [I] --------- Verifying version history table ---------
2026-08-06 10:49:35,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:49:35,868 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:49:35,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:49:36,127 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:49:36,127 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 10:49:36,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 10:49:36.351 UTC [28628] WARNING: there is no transaction in progress
2026-08-06 10:49:36,357 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 10:49:36,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:49:36,583 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 10:49:36,584 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:49:36,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 10:49:36,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 10:49:37,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0b58.c.gcp-upstream.internal','N') ;"
2026-08-06 10:49:37,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:49:37,515 [I] Table x_portal_user does not exist in database ranger
2026-08-06 10:49:37,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:49:37,756 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 10:49:37,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:49:37,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:49:38,247 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 10:49:38,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 10:49:39.153 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.168 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.184 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.198 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.214 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.230 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.281 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.287 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.300 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.310 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.319 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.332 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.341 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.347 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.354 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39.361 UTC [28737] WARNING: there is no transaction in progress
2026-08-06 10:49:39,963 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 10:49:39,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 10:49:40,202 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 10:49:40,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:49:40,460 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 10:49:40,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:49:40,701 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 10:49:40,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:49:40,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:49:41,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0b58.c.gcp-upstream.internal';"
2026-08-06 10:49:41,445 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 10:49:41,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 10:49:41,668 [I] DB_PATCHES have already been applied
I20260806 10:49:41.674433 16468 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLWW7rW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 10:49:41.674530 16468 mini_ranger.cc:193] Using host: 127.16.21.20
I20260806 10:49:41.695866 16468 mini_ranger.cc:227] Ranger admin URL: http://127.16.21.20:38217
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 10:49:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 10:49:42.873805 16468 mini_ranger.cc:160] Time spent starting Ranger: real 8.316s user 0.000s sys 0.059s
Aug 06, 2026 10:50:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 10:50:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 06, 2026 10:50:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 10:50:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 10:50:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 10:50:03.673731 16468 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/rangerkms_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/rangerkms_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.21.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.21.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.21.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/HTTP_127.16.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 10:50:03 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 10:50:03.896657 16468 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 10:50:03.925144 16468 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 10:50:04.780804 16468 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 10:50:04.780913 16468 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms
I20260806 10:50:04.780963 16468 mini_ranger_kms.cc:227] Using postgres at 127.16.21.20:39547
2026-08-06 10:50:05,155 [I] DB FLAVOR :POSTGRES
2026-08-06 10:50:05,155 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:50:05,155 [I] Checking connection
2026-08-06 10:50:05,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 10:50:05,412 [I] connection success
2026-08-06 10:50:05,412 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 10:50:05,412 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 10:50:05,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 10:50:05,645 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 10:50:05,646 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:50:05,646 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 10:50:05,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.21.20:39547/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 10:50:05,910 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 10:50:06.383867 16468 mini_ranger_kms.cc:310] Created kms service
I20260806 10:50:06.576066 16468 mini_ranger_kms.cc:326] Created kudu user
I20260806 10:50:06.621552 16468 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 10:50:06 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:06.996758 16468 mini_ranger_kms.cc:384] Added ranger policy
I20260806 10:50:06.997085 16468 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLWW7rW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLWW7rW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 10:50:06.997148 16468 mini_ranger_kms.cc:240] Using host: 127.16.21.20
I20260806 10:50:07.021838 16468 mini_ranger_kms.cc:276] Ranger KMS PID: 28917
I20260806 10:50:07.021982 16468 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.21.20:48143
Aug 06, 2026 10:50:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:50:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:50:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 10:50:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 10:50:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 10:50:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 10:50:09 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.21.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:09.319218 16468 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.423s user 0.055s sys 0.355s
I20260806 10:50:09.319507 16468 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 10:50:09.319700 16468 mini_ranger_kms.cc:401] 127.16.21.20:48143/kms/v1/keys
Aug 06 10:50:09 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.21.20@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
Aug 06 10:50:09 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.21.20@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
Aug 06 10:50:12 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:50:12.196902 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44695
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:44695
--ranger_config_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:12.535357 28970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:12.535636 28970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:12.535725 28970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:12.542920 28970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:12.543025 28970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:12.543072 28970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:12.543129 28970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:12.543179 28970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:12.556252 28970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.21.20:48143/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:44695
--ranger_config_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:12.558080 28970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:12.559464 28970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:12.571798 28975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:12.571936 28978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:12.571754 28976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:12.572619 28970 server_base.cc:1034] running on GCE node
I20260806 10:50:12.573390 28970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:12.574836 28970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:12.576035 28970 hybrid_clock.cc:648] HybridClock initialized: now 1786013412575985 us; error 49 us; skew 500 ppm
I20260806 10:50:12.576447 28970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:12 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:12.583595 28970 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:12.585130 28970 webserver.cc:460] Webserver started at http://127.16.21.62:40611/ using document root <none> and password file <none>
I20260806 10:50:12.585758 28970 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:12.585858 28970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:12.586112 28970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:50:12 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:12.642589 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "df201571b8ed4f5a85dea198574d2161"
format_stamp: "Formatted at 2026-08-06 10:50:12 on dist-test-slave-0b58"
server_key: "c45f274c980b40b0863e05f5ca6209e2"
server_key_iv: "60cca579314c565c7c2707a7f84de748"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:12.643465 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "df201571b8ed4f5a85dea198574d2161"
format_stamp: "Formatted at 2026-08-06 10:50:12 on dist-test-slave-0b58"
server_key: "c45f274c980b40b0863e05f5ca6209e2"
server_key_iv: "60cca579314c565c7c2707a7f84de748"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:12.648981 28970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:50:12.668497 28985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:12.670009 28970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:50:12.670192 28970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "df201571b8ed4f5a85dea198574d2161"
format_stamp: "Formatted at 2026-08-06 10:50:12 on dist-test-slave-0b58"
server_key: "c45f274c980b40b0863e05f5ca6209e2"
server_key_iv: "60cca579314c565c7c2707a7f84de748"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:12.670430 28970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:12.682530 28970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:12.686678 28970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:12.687004 28970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:12.707188 28970 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:44695
I20260806 10:50:12.707208 29036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:44695 every 8 connection(s)
I20260806 10:50:12.708629 28970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:12.709086 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 28970
I20260806 10:50:12.709313 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:50:12.710189 16468 external_mini_cluster.cc:1428] Setting key ee750d66b2216a9aac142fdfe04823c8
I20260806 10:50:12.714169 29037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:12.727427 29037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Bootstrap starting.
I20260806 10:50:12.730916 29037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:12.732255 29037 log.cc:824] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:50:12 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:12.735330 29037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: No bootstrap required, opened a new log
I20260806 10:50:12.745177 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:12.712852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.16.21.62:44695)
0806 10:50:12.714310 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:12.714325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:12.720920 (+ 6595us) server_negotiation.cc:407] Connection header received
0806 10:50:12.723385 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:12.723414 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:12.723956 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:12.724419 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:12.726045 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:12.728085 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:12.730038 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:12.731079 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:12.731268 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:12.736141 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:12.736151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:12.736169 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:12.736215 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:12.740376 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:12.741227 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:12.741235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:12.741243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:12.741438 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:12.741889 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:12.741894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:12.741899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:12.742656 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:12.742962 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:12.743396 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 10:50:12.743816 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":949,"threads_started":1}
I20260806 10:50:12.746023 29037 raft_consensus.cc:348] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:12.746394 29037 raft_consensus.cc:374] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:12.746510 29037 raft_consensus.cc:729] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df201571b8ed4f5a85dea198574d2161, State: Initialized, Role: FOLLOWER
I20260806 10:50:12.747195 29037 consensus_queue.cc:260] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:12.747473 29037 raft_consensus.cc:388] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:12.747630 29037 raft_consensus.cc:482] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:12.747809 29037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:12.750233 29037 raft_consensus.cc:504] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:12.750784 29037 leader_election.cc:302] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df201571b8ed4f5a85dea198574d2161; no voters:
I20260806 10:50:12.751489 29037 leader_election.cc:288] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:50:12.751591 29042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:12.754089 29037 sys_catalog.cc:564] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:12.756546 29042 raft_consensus.cc:686] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 LEADER]: Becoming Leader. State: Replica: df201571b8ed4f5a85dea198574d2161, State: Running, Role: LEADER
I20260806 10:50:12.757368 29042 consensus_queue.cc:237] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:12.763789 29043 sys_catalog.cc:455] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df201571b8ed4f5a85dea198574d2161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } } }
I20260806 10:50:12.764245 29043 sys_catalog.cc:458] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:12.764890 29044 sys_catalog.cc:455] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df201571b8ed4f5a85dea198574d2161. Latest consensus state: current_term: 1 leader_uuid: "df201571b8ed4f5a85dea198574d2161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } } }
I20260806 10:50:12.765194 29044 sys_catalog.cc:458] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:12.767876 29054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:12.777284 29054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:12.789844 29054 catalog_manager.cc:1347] Generated new cluster ID: 8923571209664698b1adb740a93a4981
I20260806 10:50:12.789984 29054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:12.801267 29037 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 10:50:12.813997 29054 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:50:12.816440 29054 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:12.834543 29054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Generated new TSK 0
I20260806 10:50:12.835397 29054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:50:13.636466 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44695
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:50:13 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013413, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 10:50:14.033776 29083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:14.034080 29083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:14.034263 29083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:14.041836 29083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:14.042009 29083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:14.042223 29083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:50:14.054140 29083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.21.20:48143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:14.055495 29083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:14.056818 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:14.080467 29093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:14.082412 29094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:14.086508 29096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:14.087590 29083 server_base.cc:1034] running on GCE node
I20260806 10:50:14.088240 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:14.089296 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:14.090595 29083 hybrid_clock.cc:648] HybridClock initialized: now 1786013414090505 us; error 72 us; skew 500 ppm
I20260806 10:50:14.090999 29083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:14.094357 29083 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:50:14.097630 29083 webserver.cc:460] Webserver started at http://127.16.21.1:41301/ using document root <none> and password file <none>
I20260806 10:50:14.098512 29083 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:14.098655 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:14.098990 29083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:14.126567 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "2e344caa892f4f59bc29e9fdbddeaec2"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "bd6b922946c10a1af6b2fc41dcce5161"
server_key_iv: "ee2c22f441716760d07759d83d9e680d"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:14.127779 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "2e344caa892f4f59bc29e9fdbddeaec2"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "bd6b922946c10a1af6b2fc41dcce5161"
server_key_iv: "ee2c22f441716760d07759d83d9e680d"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:14.135548 29083 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260806 10:50:14.157317 29103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:14.159219 29083 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 10:50:14.159420 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "2e344caa892f4f59bc29e9fdbddeaec2"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "bd6b922946c10a1af6b2fc41dcce5161"
server_key_iv: "ee2c22f441716760d07759d83d9e680d"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:14.159643 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:14.182811 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:14.186874 29083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:14.187181 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:14.188197 29083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:14.189754 29083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:14.189828 29083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:14.189960 29083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:14.190011 29083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:14.259317 29083 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:35679
I20260806 10:50:14.259375 29215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:35679 every 8 connection(s)
I20260806 10:50:14.261209 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:50:14.263273 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29083
I20260806 10:50:14.263545 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:50:14.264570 16468 external_mini_cluster.cc:1428] Setting key 9741b8036ceb2030dc98d66bf6e47b4b
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:14.306689 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:14.265323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44695 (local address 127.16.21.1:54557)
0806 10:50:14.278589 (+ 13266us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:14.278646 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:14.281419 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:14.283678 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:14.283689 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:14.284419 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:14.285366 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:14.285391 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:14.287134 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:14.287139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:14.289155 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:14.289170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:14.290443 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:14.290449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:14.290736 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:14.291521 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:14.291554 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:14.297018 (+ 5464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:14.301969 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:14.301978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:14.301994 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:14.302595 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:14.303066 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:14.303071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:14.303075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:14.303352 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:14.304089 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:14.304097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:14.304520 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 10:50:14.304816 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 10:50:14.305326 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12942,"thread_start_us":12890,"threads_started":1}
I20260806 10:50:14.308068 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:14.264545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:54557 (local address 127.16.21.62:44695)
0806 10:50:14.282987 (+ 18442us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:14.282993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:14.283014 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:50:14.283072 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:14.283078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:14.283175 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:14.285599 (+ 2424us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:14.285799 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:14.286947 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:14.289356 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:14.290028 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:14.290142 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:14.297288 (+ 7146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:14.297295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:14.297299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:14.297348 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:14.301794 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:14.302754 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:14.302760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:14.302763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:14.302922 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:14.303494 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:14.303500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:14.303503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:14.303861 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:14.304004 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:14.307416 (+ 3412us) server_negotiation.cc:299] Negotiation successful
0806 10:50:14.307682 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18315,"thread_start_us":18716,"threads_started":1}
I20260806 10:50:14.309774 29216 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44695
I20260806 10:50:14.310099 29216 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:14.310953 29216 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:14.313385 29002 ts_manager.cc:194] Registered new tserver with Master: 2e344caa892f4f59bc29e9fdbddeaec2 (127.16.21.1:35679)
I20260806 10:50:14.315438 29002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:54557
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:50:14.371515 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44695
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013413, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013413, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
W20260806 10:50:14.865520 29222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:14.866014 29222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:14.866222 29222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:14.878368 29222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:14.878577 29222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:14.878831 29222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:50:14.894778 29222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.21.20:48143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:14.896646 29222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:14.898429 29222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:14.928740 29229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:14.930384 29230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:14.948820 29222 server_base.cc:1034] running on GCE node
W20260806 10:50:14.948647 29232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:14.949884 29222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:14.953784 29222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:14.955175 29222 hybrid_clock.cc:648] HybridClock initialized: now 1786013414955054 us; error 109 us; skew 500 ppm
I20260806 10:50:14.955624 29222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:14.959841 29222 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:50:14.963611 29222 webserver.cc:460] Webserver started at http://127.16.21.2:43425/ using document root <none> and password file <none>
I20260806 10:50:14.964586 29222 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:14.964788 29222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:14.965193 29222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:50:14 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:14.989220 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "2f14232e035f46078aa184252eab5b9f"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "c056e4f35dccd6d2b558bf00027b785d"
server_key_iv: "867e3754641aefac2c14d22cd1db97c6"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:14.993005 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "2f14232e035f46078aa184252eab5b9f"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "c056e4f35dccd6d2b558bf00027b785d"
server_key_iv: "867e3754641aefac2c14d22cd1db97c6"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:14.999475 29222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 10:50:15.018054 29239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:15.019688 29222 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 10:50:15.019892 29222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "2f14232e035f46078aa184252eab5b9f"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "c056e4f35dccd6d2b558bf00027b785d"
server_key_iv: "867e3754641aefac2c14d22cd1db97c6"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:15.020138 29222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:15.075217 29222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:15.079380 29222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:15.079761 29222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:15.080893 29222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:15.082278 29222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:15.082387 29222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:15.082530 29222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:15.082602 29222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:15.123170 29222 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:37685
I20260806 10:50:15.123193 29359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:37685 every 8 connection(s)
I20260806 10:50:15.124487 29222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:50:15.130698 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29222
I20260806 10:50:15.130882 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:50:15.131461 16468 external_mini_cluster.cc:1428] Setting key ea7cced977e6fcf89f72952a28515277
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:15.148550 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.127193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58391 (local address 127.16.21.62:44695)
0806 10:50:15.127734 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:15.127740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:15.128683 (+ 943us) server_negotiation.cc:407] Connection header received
0806 10:50:15.130573 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:15.130579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:15.130693 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:15.130816 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:15.132370 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.133280 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.135696 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.136350 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.136480 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.140975 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:15.140983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:15.140991 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:15.141035 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:15.144879 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.145730 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.145739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.145745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.145893 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.146439 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.146447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.146453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.146839 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:15.147009 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:15.147798 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 10:50:15.148051 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":269,"threads_started":1}
I20260806 10:50:15.149611 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.127640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44695 (local address 127.16.21.2:58391)
0806 10:50:15.128506 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:15.128558 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:15.130316 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:15.131044 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:15.131052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:15.131473 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:15.131998 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:15.132042 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.133500 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.133508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:15.135524 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:15.135543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.136591 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.136601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:15.136917 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.137769 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:15.137801 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:15.140737 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:15.145020 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:15.145029 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:15.145045 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:15.145600 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:15.146037 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:15.146043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:15.146047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:15.146313 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:15.146991 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:15.146999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:15.147375 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 10:50:15.147673 (+ 298us) client_negotiation.cc:240] Negotiation successful
0806 10:50:15.148156 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":354,"threads_started":1}
I20260806 10:50:15.151104 29360 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44695
I20260806 10:50:15.151429 29360 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:15.152351 29360 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:15.154047 29002 ts_manager.cc:194] Registered new tserver with Master: 2f14232e035f46078aa184252eab5b9f (127.16.21.2:37685)
I20260806 10:50:15.155169 29002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:58391
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:50:15.203213 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44695
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:15.318825 29216 heartbeater.cc:503] Master 127.16.21.62:44695 was elected leader, sending a full tablet report...
W20260806 10:50:15.537236 29366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:15.537508 29366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:15.537691 29366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:15.544960 29366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:15.545089 29366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:15.545276 29366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:50:15.555028 29366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.21.20:48143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:15.556499 29366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:15.557845 29366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:15.571559 29371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:15.571481 29374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:15.571537 29372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:15.572798 29366 server_base.cc:1034] running on GCE node
I20260806 10:50:15.573484 29366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:15.574479 29366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:15.575706 29366 hybrid_clock.cc:648] HybridClock initialized: now 1786013415575648 us; error 47 us; skew 500 ppm
I20260806 10:50:15.576035 29366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013415, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:15.579136 29366 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:50:15.580651 29366 webserver.cc:460] Webserver started at http://127.16.21.3:44827/ using document root <none> and password file <none>
I20260806 10:50:15.581383 29366 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:15.581526 29366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:15.581876 29366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013415, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:15.596468 29366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "6f31ca2385f449d3bbb73543991bcab4"
format_stamp: "Formatted at 2026-08-06 10:50:15 on dist-test-slave-0b58"
server_key: "38e38ba59739acdd5335ef16211aac9d"
server_key_iv: "84841aa6a213fcdfe60a87dffb2f6f44"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:15.597338 29366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "6f31ca2385f449d3bbb73543991bcab4"
format_stamp: "Formatted at 2026-08-06 10:50:15 on dist-test-slave-0b58"
server_key: "38e38ba59739acdd5335ef16211aac9d"
server_key_iv: "84841aa6a213fcdfe60a87dffb2f6f44"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:15.602953 29366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 10:50:15.624887 29381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:15.626639 29366 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.005s
I20260806 10:50:15.626852 29366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "6f31ca2385f449d3bbb73543991bcab4"
format_stamp: "Formatted at 2026-08-06 10:50:15 on dist-test-slave-0b58"
server_key: "38e38ba59739acdd5335ef16211aac9d"
server_key_iv: "84841aa6a213fcdfe60a87dffb2f6f44"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:15.627097 29366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:15.650121 29366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:15.654429 29366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:15.654822 29366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:15.655764 29366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:15.657433 29366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:15.657536 29366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:15.657682 29366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:15.657758 29366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:15.693257 29366 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:39903
I20260806 10:50:15.693305 29494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:39903 every 8 connection(s)
I20260806 10:50:15.694568 29366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:50:15.702099 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29366
I20260806 10:50:15.702329 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:50:15.702780 16468 external_mini_cluster.cc:1428] Setting key 12c9a18fbd1386f7791fc53c0b3086b7
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013415, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:15.716436 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.697440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:54637 (local address 127.16.21.62:44695)
0806 10:50:15.697889 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:15.697894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:15.698983 (+ 1089us) server_negotiation.cc:407] Connection header received
0806 10:50:15.700952 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:15.700958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:15.701073 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:15.701194 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:15.702746 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.703673 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.705548 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.706113 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.706237 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.709651 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:15.709658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:15.709663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:15.709703 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:15.713272 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.713988 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.713993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.713998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.714147 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.714555 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.714560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.714564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.714915 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:15.715052 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:15.715820 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 10:50:15.716046 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260806 10:50:15.717675 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.698071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44695 (local address 127.16.21.3:54637)
0806 10:50:15.698823 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:15.698860 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:15.700687 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:15.701437 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:15.701445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:15.701857 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:15.702576 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:15.702595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.703876 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.703880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:15.705405 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:15.705417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.706305 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.706310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:15.706505 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.707218 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:15.707244 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:15.709448 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:15.713412 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:15.713418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:15.713430 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:15.713846 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:15.714259 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:15.714262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:15.714264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:15.714448 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:15.715025 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:15.715030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:15.715432 (+ 402us) client_negotiation.cc:769] Sending connection context
0806 10:50:15.715681 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 10:50:15.716035 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":326,"threads_started":1}
I20260806 10:50:15.719033 29495 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44695
I20260806 10:50:15.719287 29495 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:15.719908 29495 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:15.721210 29002 ts_manager.cc:194] Registered new tserver with Master: 6f31ca2385f449d3bbb73543991bcab4 (127.16.21.3:39903)
I20260806 10:50:15.721958 29002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:54637
I20260806 10:50:15.730342 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:50:15.732355 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29083
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:50:15.748406 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.733212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.16.21.1:35679)
0806 10:50:15.733825 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:15.733834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:15.733876 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:50:15.733951 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:15.733957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:15.734136 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:15.734294 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:15.735734 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.736800 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.738335 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.738899 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.739035 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.743316 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:15.743327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:15.743341 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:15.743373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:15.746009 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.746662 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.746668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.746670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.746797 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.747107 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.747110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.747112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.747413 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:15.747639 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:15.747879 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 10:50:15.748065 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":267,"threads_started":1}
I20260806 10:50:15.749595 29211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:15.838042 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29083
I20260806 10:50:15.856139 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29222
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:50:15.873121 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.857089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.16.21.2:37685)
0806 10:50:15.857697 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:15.857704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:15.857730 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:50:15.857800 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:15.857806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:15.858084 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:15.858227 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:15.859480 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.860660 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.862265 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.863000 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.863148 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.867659 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:15.867670 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:15.867683 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:15.867716 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:15.870281 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.870962 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.870969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.870972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.871110 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.871508 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.871513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.871518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.871852 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:15.872148 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:15.872492 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 10:50:15.872763 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":330,"threads_started":1}
I20260806 10:50:15.874461 29355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:15.958465 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29222
I20260806 10:50:15.975975 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29366
Aug 06 10:50:15 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:50:15.992121 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:15.976962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.16.21.3:39903)
0806 10:50:15.977694 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:15.977704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:15.977732 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:50:15.977800 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:15.977807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:15.978020 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:15.978147 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:15.979446 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.980603 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.981896 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:15.982467 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:15.982563 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:15.987024 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:15.987034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:15.987047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:15.987078 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:15.989720 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.990361 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.990367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.990369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.990490 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:15.990808 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:15.990812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:15.990813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:15.991125 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:15.991373 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:15.991544 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 10:50:15.991725 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":416,"threads_started":1}
I20260806 10:50:15.993505 29490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:16.090698 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29366
I20260806 10:50:16.107750 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 28970
I20260806 10:50:16.109004 29032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:16.216081 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 28970
I20260806 10:50:16.245661 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44695
--webserver_interface=127.16.21.62
--webserver_port=40611
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:44695
--ranger_config_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:16.569398 29507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:16.569718 29507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:16.569890 29507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:16.576941 29507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:16.577085 29507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:16.577214 29507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:16.577310 29507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:16.577404 29507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:16.587394 29507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:44695
--ranger_config_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=40611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:16.588653 29507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:16.589972 29507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:16.601689 29513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:16.601791 29512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:16.601791 29515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:16.602864 29507 server_base.cc:1034] running on GCE node
I20260806 10:50:16.603612 29507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:16.605019 29507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:16.606223 29507 hybrid_clock.cc:648] HybridClock initialized: now 1786013416606161 us; error 59 us; skew 500 ppm
I20260806 10:50:16.606565 29507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:16 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013416, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:16.609676 29507 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:16.611106 29507 webserver.cc:460] Webserver started at http://127.16.21.62:40611/ using document root <none> and password file <none>
I20260806 10:50:16.611712 29507 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:16.611850 29507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:50:16 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013416, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:16.693909 29507 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:50:16.696836 29523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:16.698118 29507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:16.698331 29507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "df201571b8ed4f5a85dea198574d2161"
format_stamp: "Formatted at 2026-08-06 10:50:12 on dist-test-slave-0b58"
server_key: "c45f274c980b40b0863e05f5ca6209e2"
server_key_iv: "60cca579314c565c7c2707a7f84de748"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:16.699295 29507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:16.727897 29507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:16.731398 29507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:16.731690 29507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:16.750022 29507 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:44695
I20260806 10:50:16.750058 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:44695 every 8 connection(s)
I20260806 10:50:16.751451 29507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:16.755388 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29507
I20260806 10:50:16.756492 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:35679
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:44695
--webserver_port=41301
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:16.759485 29575 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:50:16.764706 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Bootstrap starting.
I20260806 10:50:16.785461 29575 log.cc:824] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:16.794579 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:50:16.795116 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Bootstrap complete.
I20260806 10:50:16.804764 29575 raft_consensus.cc:348] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:16.805611 29575 raft_consensus.cc:729] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df201571b8ed4f5a85dea198574d2161, State: Initialized, Role: FOLLOWER
I20260806 10:50:16.806254 29575 consensus_queue.cc:260] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:16.806494 29575 raft_consensus.cc:388] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:16.806633 29575 raft_consensus.cc:482] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:16.806814 29575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:50:16.810858 29575 raft_consensus.cc:504] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:16.811409 29575 leader_election.cc:302] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df201571b8ed4f5a85dea198574d2161; no voters:
I20260806 10:50:16.812147 29575 leader_election.cc:288] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:50:16.812311 29579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:50:16.814522 29575 sys_catalog.cc:564] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:16.819480 29579 raft_consensus.cc:686] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [term 2 LEADER]: Becoming Leader. State: Replica: df201571b8ed4f5a85dea198574d2161, State: Running, Role: LEADER
I20260806 10:50:16.820171 29579 consensus_queue.cc:237] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } }
I20260806 10:50:16.823105 29580 sys_catalog.cc:455] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df201571b8ed4f5a85dea198574d2161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } } }
I20260806 10:50:16.823582 29580 sys_catalog.cc:458] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:16.824205 29581 sys_catalog.cc:455] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df201571b8ed4f5a85dea198574d2161. Latest consensus state: current_term: 2 leader_uuid: "df201571b8ed4f5a85dea198574d2161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df201571b8ed4f5a85dea198574d2161" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44695 } } }
I20260806 10:50:16.824518 29581 sys_catalog.cc:458] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:16.834077 29590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:16.841872 29590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:16.843842 29590 catalog_manager.cc:1259] Loaded cluster ID: 8923571209664698b1adb740a93a4981
I20260806 10:50:16.844061 29590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:16.849217 29590 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:16.851109 29590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Loaded TSK: 0
I20260806 10:50:16.853029 29590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:50:16.855581 29575 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 10:50:17.216235 29577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:17.216627 29577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:17.216816 29577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:17.228560 29577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:17.228807 29577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:17.229061 29577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:50:17.241309 29577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:35679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=41301
--tserver_master_addrs=127.16.21.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:17.242925 29577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:17.244554 29577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:17.268698 29614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:17.268934 29613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:17.290119 29616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:17.310137 29577 server_base.cc:1034] running on GCE node
I20260806 10:50:17.313069 29577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:17.320667 29577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:17.321991 29577 hybrid_clock.cc:648] HybridClock initialized: now 1786013417321915 us; error 69 us; skew 500 ppm
I20260806 10:50:17.322394 29577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:17 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:17.326792 29577 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:50:17.348667 29577 webserver.cc:460] Webserver started at http://127.16.21.1:41301/ using document root <none> and password file <none>
I20260806 10:50:17.349745 29577 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:17.349992 29577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:50:17 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:17.397481 29577 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.007s sys 0.004s
I20260806 10:50:17.402081 29625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:17.403980 29577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:50:17.404160 29577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "2e344caa892f4f59bc29e9fdbddeaec2"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "bd6b922946c10a1af6b2fc41dcce5161"
server_key_iv: "ee2c22f441716760d07759d83d9e680d"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:17.405261 29577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:17.441841 29577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:17.446367 29577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:17.446663 29577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:17.447830 29577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:17.449442 29577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:17.449545 29577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:17.449640 29577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:17.449707 29577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:17.511893 29577 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:35679
I20260806 10:50:17.511953 29737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:35679 every 8 connection(s)
I20260806 10:50:17.513804 29577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:50:17.524744 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29577
I20260806 10:50:17.525602 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:37685
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:44695
--webserver_port=43425
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:50:17 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013417, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:17.546077 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:17.517446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44695 (local address 127.16.21.1:42543)
0806 10:50:17.522082 (+ 4636us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:17.522135 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:17.524585 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:17.526618 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:17.526628 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:17.527243 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:17.528009 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:17.528035 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:17.530400 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:17.530405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:17.532407 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:17.532424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:17.533893 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:17.533903 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:17.534205 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:17.535012 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:17.535045 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:17.537804 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:17.541839 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:17.541849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:17.541862 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:17.542254 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:17.542705 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:17.542713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:17.542717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:17.542914 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:17.543633 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:17.543642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:17.543946 (+ 304us) client_negotiation.cc:769] Sending connection context
0806 10:50:17.544204 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 10:50:17.544766 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4355,"thread_start_us":4145,"threads_started":1}
I20260806 10:50:17.546075 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:17.518576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:42543 (local address 127.16.21.62:44695)
0806 10:50:17.519438 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:17.519462 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:17.522280 (+ 2818us) server_negotiation.cc:407] Connection header received
0806 10:50:17.525295 (+ 3015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:17.525314 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:17.525772 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:17.526441 (+ 669us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:17.528242 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:17.530207 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:17.532562 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:17.533632 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:17.533829 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:17.538084 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:17.538094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:17.538112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:17.538157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:17.541677 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:17.542421 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:17.542427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:17.542432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:17.542585 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:17.543043 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:17.543047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:17.543049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:17.543488 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:17.543707 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:17.544454 (+ 747us) server_negotiation.cc:299] Negotiation successful
0806 10:50:17.544841 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":309,"threads_started":1}
I20260806 10:50:17.549736 29738 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44695
I20260806 10:50:17.550043 29738 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:17.550863 29738 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
W20260806 10:50:17.554709 29738 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:44695 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 10:50:17.554965 29738 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:17.555222 29738 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:17.556638 29738 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:17.556957 29738 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
W20260806 10:50:17.563171 29738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:50:17.941753 29743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:17.942096 29743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:17.942314 29743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:17.952366 29743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:17.952533 29743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:17.952744 29743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:50:17.964915 29743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:37685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=43425
--tserver_master_addrs=127.16.21.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:17.966717 29743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:17.968441 29743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:17.994172 29757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:17.996575 29758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:18.002403 29762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:18.002905 29743 server_base.cc:1034] running on GCE node
I20260806 10:50:18.003780 29743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:18.020514 29743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:18.021837 29743 hybrid_clock.cc:648] HybridClock initialized: now 1786013418021769 us; error 61 us; skew 500 ppm
I20260806 10:50:18.022327 29743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:18.030505 29743 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:50:18.032578 29743 webserver.cc:460] Webserver started at http://127.16.21.2:43425/ using document root <none> and password file <none>
I20260806 10:50:18.033532 29743 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:18.033699 29743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:18.061494 29743 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.004s
I20260806 10:50:18.065171 29770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:18.066910 29743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:50:18.067080 29743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "2f14232e035f46078aa184252eab5b9f"
format_stamp: "Formatted at 2026-08-06 10:50:14 on dist-test-slave-0b58"
server_key: "c056e4f35dccd6d2b558bf00027b785d"
server_key_iv: "867e3754641aefac2c14d22cd1db97c6"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:18.068573 29743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:18.100220 29743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:18.105253 29743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:18.105602 29743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:18.112982 29743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:18.114778 29743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:18.115263 29743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:18.115437 29743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:18.115551 29743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:18.169934 29743 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:37685
I20260806 10:50:18.170002 29883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:37685 every 8 connection(s)
I20260806 10:50:18.171790 29743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:50:18.178180 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29743
I20260806 10:50:18.179289 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.21.20:48143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:39903
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:44695
--webserver_port=44827
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:18.215266 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:18.175354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:38797 (local address 127.16.21.62:44695)
0806 10:50:18.191536 (+ 16182us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:18.191541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:18.192280 (+ 739us) server_negotiation.cc:407] Connection header received
0806 10:50:18.195171 (+ 2891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:18.195177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:18.195272 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:18.195399 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:18.197576 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:18.198828 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:18.201151 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:18.201943 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:18.202081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:18.206203 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:18.206213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:18.206221 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:18.206267 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:18.210462 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:18.211427 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:18.211435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:18.211441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:18.211588 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:18.212137 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:18.212145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:18.212151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:18.212595 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:18.212761 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:18.214502 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0806 10:50:18.214772 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16068,"thread_start_us":16167,"threads_started":1}
I20260806 10:50:18.217190 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:18.175919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44695 (local address 127.16.21.2:38797)
0806 10:50:18.192054 (+ 16135us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:18.192098 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:18.194749 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:18.195858 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:18.195870 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:18.196572 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:18.197358 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:18.197382 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:18.199063 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:18.199070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:18.200969 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:18.200987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:18.202206 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:18.202214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:18.202483 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:18.203284 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:18.203314 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:18.205906 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:18.210627 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:18.210638 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:18.210656 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:18.211280 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:18.211716 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:18.211724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:18.211730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:18.211975 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:18.212834 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:18.212845 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:18.213265 (+ 420us) client_negotiation.cc:769] Sending connection context
0806 10:50:18.213561 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 10:50:18.214117 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15907,"thread_start_us":15709,"threads_started":1}
I20260806 10:50:18.219345 29884 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44695
I20260806 10:50:18.219661 29884 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:18.220633 29884 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:18.222924 29540 ts_manager.cc:194] Registered new tserver with Master: 2f14232e035f46078aa184252eab5b9f (127.16.21.2:37685)
I20260806 10:50:18.225317 29540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:38797
I20260806 10:50:18.563334 29738 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:18.563791 29738 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:18.565104 29540 ts_manager.cc:194] Registered new tserver with Master: 2e344caa892f4f59bc29e9fdbddeaec2 (127.16.21.1:35679)
I20260806 10:50:18.566293 29540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:42543
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
W20260806 10:50:18.606218 29888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:18.606566 29888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:18.606778 29888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:18.617955 29888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:18.618151 29888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:18.618374 29888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:50:18.633409 29888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.21.20:48143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:39903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=44827
--tserver_master_addrs=127.16.21.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:18.635149 29888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:18.636788 29888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:18.679724 29894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:18.680073 29893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:18.685511 29888 server_base.cc:1034] running on GCE node
W20260806 10:50:18.685827 29896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:18.686647 29888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:18.687804 29888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:18.689069 29888 hybrid_clock.cc:648] HybridClock initialized: now 1786013418689011 us; error 43 us; skew 500 ppm
I20260806 10:50:18.689452 29888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:18.693578 29888 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:50:18.697813 29888 webserver.cc:460] Webserver started at http://127.16.21.3:44827/ using document root <none> and password file <none>
I20260806 10:50:18.698745 29888 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:18.698904 29888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:50:18 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:18.722643 29888 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:50:18.726238 29904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:18.727803 29888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:50:18.727995 29888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "6f31ca2385f449d3bbb73543991bcab4"
format_stamp: "Formatted at 2026-08-06 10:50:15 on dist-test-slave-0b58"
server_key: "38e38ba59739acdd5335ef16211aac9d"
server_key_iv: "84841aa6a213fcdfe60a87dffb2f6f44"
server_key_version: "kuduclusterkey@0"
I20260806 10:50:18.729074 29888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:18.770058 29888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:18.774722 29888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:18.775022 29888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:18.776113 29888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:18.777601 29888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:18.777729 29888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:18.777874 29888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:18.777964 29888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:50:19 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for HTTP/127.16.21.20@KRBTEST.COM
I20260806 10:50:19.210217 29888 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:39903
I20260806 10:50:19.210309 30017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:39903 every 8 connection(s)
I20260806 10:50:19.212173 29888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:50:19.220849 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 29888
I20260806 10:50:19.230046 29884 heartbeater.cc:503] Master 127.16.21.62:44695 was elected leader, sending a full tablet report...
Aug 06 10:50:19 dist-test-slave-0b58 krb5kdc[28522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013418, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:19.260516 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.218446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:58247 (local address 127.16.21.62:44695)
0806 10:50:19.233917 (+ 15471us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.233923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.233945 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:50:19.236370 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.236376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.236495 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.236588 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:19.239008 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.240145 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.242550 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.243238 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.243334 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.250379 (+ 7045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:19.250385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:19.250390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:19.250438 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:19.256029 (+ 5591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.256981 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.256987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.256991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.257743 (+ 752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.258326 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.258331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.258334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.258684 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:19.258812 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.259838 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.260081 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15353,"thread_start_us":15267,"threads_started":1}
I20260806 10:50:19.262939 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.217903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44695 (local address 127.16.21.3:58247)
0806 10:50:19.219347 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:19.219405 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:19.235964 (+ 16559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:19.237306 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:19.237317 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:19.237973 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:19.238794 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.238820 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.240387 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.240392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.242379 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.242394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.243679 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.243687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.243973 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.245188 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:19.245218 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:19.250127 (+ 4909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:19.256191 (+ 6064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:19.256200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:19.256216 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:19.256838 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:19.257913 (+ 1075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:19.257918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:19.257922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:19.258187 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:19.258926 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:19.258935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:19.259370 (+ 435us) client_negotiation.cc:769] Sending connection context
0806 10:50:19.261061 (+ 1691us) client_negotiation.cc:240] Negotiation successful
0806 10:50:19.261583 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":931,"threads_started":1}
I20260806 10:50:19.264830 30018 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44695
I20260806 10:50:19.265159 30018 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:19.266047 30018 heartbeater.cc:511] Master 127.16.21.62:44695 requested a full tablet report, sending...
I20260806 10:50:19.267735 29540 ts_manager.cc:194] Registered new tserver with Master: 6f31ca2385f449d3bbb73543991bcab4 (127.16.21.3:39903)
I20260806 10:50:19.268923 29540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:58247
I20260806 10:50:19.275345 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.233962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.16.21.62:44695)
0806 10:50:19.244758 (+ 10796us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.244764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.244782 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:50:19.244830 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.244835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.244921 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.246056 (+ 1135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:19.250833 (+ 4777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.252160 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.256390 (+ 4230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.257076 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.257239 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.260763 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:19.260770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:19.260775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:19.260812 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:19.267248 (+ 6436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.269379 (+ 2131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.269384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.269388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.269533 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.272289 (+ 2756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.272294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.272298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.272794 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:19.274545 (+ 1751us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.274686 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.274987 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10644,"thread_start_us":10496,"threads_started":1}
I20260806 10:50:19.278486 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:50:19.316679 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.296946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.16.21.62:44695)
0806 10:50:19.297132 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.297138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.299761 (+ 2623us) server_negotiation.cc:407] Connection header received
0806 10:50:19.299932 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.299942 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.300038 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.300152 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:19.302268 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.303314 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.305404 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.306047 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.306144 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.309749 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:19.309755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:19.309760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:19.309797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:19.313519 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.314350 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.314356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.314359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.314524 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.314994 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.314999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.315003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.315427 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:19.315703 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.316075 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.316358 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:50:19.323479 29540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:50:19.335124 29540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:50:19.406241 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.355823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:39903 (local address 127.0.0.1:33840)
0806 10:50:19.391737 (+ 35914us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:19.391793 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:19.391986 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:19.396977 (+ 4991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:19.396988 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:19.397044 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:19.397668 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.397684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.399216 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.399220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.401894 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.401907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.405300 (+ 3393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.405307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.405502 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.405536 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:50:19.405624 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 10:50:19.405758 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35613,"mutex_wait_us":3,"thread_start_us":38654,"threads_started":1}
I20260806 10:50:19.406708 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.356432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.16.21.2:37685)
0806 10:50:19.394096 (+ 37664us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.394104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.394992 (+ 888us) server_negotiation.cc:407] Connection header received
0806 10:50:19.395142 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.395150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.395397 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.395610 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:19.398465 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.400023 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.402773 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.404392 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.404567 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.404834 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.405893 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.406060 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37505,"thread_start_us":37719,"threads_started":1}
I20260806 10:50:19.407207 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.356517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.16.21.3:39903)
0806 10:50:19.396355 (+ 39838us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.396363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.396397 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:50:19.396491 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.396498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.396703 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.396917 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:19.397921 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.398997 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.402068 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.403523 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.404910 (+ 1387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.405137 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.406689 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.406863 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39645,"thread_start_us":39506,"threads_started":1}
I20260806 10:50:19.407579 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.356098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:37685 (local address 127.0.0.1:56904)
0806 10:50:19.394749 (+ 38651us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:19.394767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:19.394880 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:19.397217 (+ 2337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:19.397222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:19.397254 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:19.398266 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.398280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.400206 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.400211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.402573 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.402588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.405025 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.405033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.405251 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.405321 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 10:50:19.405453 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:50:19.405640 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38517,"mutex_wait_us":664,"thread_start_us":41327,"threads_started":1}
I20260806 10:50:19.409579 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.357615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.16.21.1:35679)
0806 10:50:19.393491 (+ 35876us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.393500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.397867 (+ 4367us) server_negotiation.cc:407] Connection header received
0806 10:50:19.400855 (+ 2988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.400865 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.401070 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.401266 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:19.402643 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.404115 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.406714 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.408114 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.408255 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.408507 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.409080 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.409245 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35699,"thread_start_us":35887,"threads_started":1}
I20260806 10:50:19.410184 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.357014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:35679 (local address 127.0.0.1:45856)
0806 10:50:19.397749 (+ 40735us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:19.397768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:19.400639 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:19.401556 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:19.401560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:19.401991 (+ 431us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:19.402406 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.402419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.404365 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.404370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.406525 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.406545 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.408703 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.408712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.408870 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.408908 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:50:19.409756 (+ 848us) client_negotiation.cc:240] Negotiation successful
0806 10:50:19.409876 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40582,"thread_start_us":35274,"threads_started":1}
I20260806 10:50:19.411500 29818 tablet_service.cc:1467] Processing CreateTablet for tablet afbede0b81e04e988a3e3d5b1e468904 (DEFAULT_TABLE table=test-table [id=208292c31209490ebef8efe2a8fa25ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:19.411511 29953 tablet_service.cc:1467] Processing CreateTablet for tablet afbede0b81e04e988a3e3d5b1e468904 (DEFAULT_TABLE table=test-table [id=208292c31209490ebef8efe2a8fa25ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:19.413141 29818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbede0b81e04e988a3e3d5b1e468904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:19.413434 29953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbede0b81e04e988a3e3d5b1e468904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:19.414405 29673 tablet_service.cc:1467] Processing CreateTablet for tablet afbede0b81e04e988a3e3d5b1e468904 (DEFAULT_TABLE table=test-table [id=208292c31209490ebef8efe2a8fa25ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:19.415974 29673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbede0b81e04e988a3e3d5b1e468904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:19.434170 30046 tablet_bootstrap.cc:492] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Bootstrap starting.
I20260806 10:50:19.439464 30046 tablet_bootstrap.cc:654] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:19.440244 30047 tablet_bootstrap.cc:492] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Bootstrap starting.
I20260806 10:50:19.442203 30046 log.cc:824] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:19.443868 30047 tablet_bootstrap.cc:654] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:19.444900 30048 tablet_bootstrap.cc:492] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Bootstrap starting.
I20260806 10:50:19.445214 30047 log.cc:824] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:19.448987 30048 tablet_bootstrap.cc:654] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:19.450521 30048 log.cc:824] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:19.455235 30047 tablet_bootstrap.cc:492] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: No bootstrap required, opened a new log
I20260806 10:50:19.455628 30047 ts_tablet_manager.cc:1397] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260806 10:50:19.458365 30046 tablet_bootstrap.cc:492] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: No bootstrap required, opened a new log
I20260806 10:50:19.458645 30046 ts_tablet_manager.cc:1397] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.012s
I20260806 10:50:19.459570 30048 tablet_bootstrap.cc:492] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: No bootstrap required, opened a new log
I20260806 10:50:19.460189 30048 ts_tablet_manager.cc:1397] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260806 10:50:19.466670 30047 raft_consensus.cc:348] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.467262 30047 raft_consensus.cc:374] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:19.467406 30047 raft_consensus.cc:729] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f31ca2385f449d3bbb73543991bcab4, State: Initialized, Role: FOLLOWER
I20260806 10:50:19.468160 30047 consensus_queue.cc:260] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.469726 30046 raft_consensus.cc:348] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.470175 30046 raft_consensus.cc:374] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:19.470305 30046 raft_consensus.cc:729] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f14232e035f46078aa184252eab5b9f, State: Initialized, Role: FOLLOWER
I20260806 10:50:19.470899 30046 consensus_queue.cc:260] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.471547 30048 raft_consensus.cc:348] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.472103 30048 raft_consensus.cc:374] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:19.472298 30048 raft_consensus.cc:729] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e344caa892f4f59bc29e9fdbddeaec2, State: Initialized, Role: FOLLOWER
I20260806 10:50:19.473088 30048 consensus_queue.cc:260] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.476940 30047 ts_tablet_manager.cc:1428] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Time spent starting tablet: real 0.021s user 0.019s sys 0.000s
I20260806 10:50:19.477319 30018 heartbeater.cc:503] Master 127.16.21.62:44695 was elected leader, sending a full tablet report...
I20260806 10:50:19.478083 30048 ts_tablet_manager.cc:1428] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Time spent starting tablet: real 0.018s user 0.005s sys 0.011s
I20260806 10:50:19.478387 29738 heartbeater.cc:503] Master 127.16.21.62:44695 was elected leader, sending a full tablet report...
I20260806 10:50:19.481230 30046 ts_tablet_manager.cc:1428] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Time spent starting tablet: real 0.022s user 0.011s sys 0.007s
W20260806 10:50:19.484710 30019 tablet.cc:2369] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:50:19.522809 29739 tablet.cc:2369] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:19.594455 30053 raft_consensus.cc:482] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:50:19.594774 30053 raft_consensus.cc:504] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.596593 30053 leader_election.cc:288] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f31ca2385f449d3bbb73543991bcab4 (127.16.21.3:39903), 2e344caa892f4f59bc29e9fdbddeaec2 (127.16.21.1:35679)
I20260806 10:50:19.605049 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.597137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:54085 (local address 127.16.21.3:39903)
0806 10:50:19.597378 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.597386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.598083 (+ 697us) server_negotiation.cc:407] Connection header received
0806 10:50:19.598450 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.598457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.598544 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.598598 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:19.599488 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.600555 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.603007 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.604122 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.604259 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.604388 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.604691 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.604814 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:50:19.605609 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.597708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:59909 (local address 127.16.21.1:35679)
0806 10:50:19.597913 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.597919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.598715 (+ 796us) server_negotiation.cc:407] Connection header received
0806 10:50:19.598927 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.598932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.599013 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.599111 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:19.599836 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.600890 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.603094 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.604294 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.604447 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.604568 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.605197 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.605347 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:50:19.605753 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.596805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:39903 (local address 127.16.21.2:54085)
0806 10:50:19.597952 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:19.597977 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:19.598235 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:19.598896 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:19.598901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:19.598927 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:19.599334 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.599346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.600705 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.600709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.602817 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.602843 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.604341 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.604346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.604503 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.604536 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:50:19.605219 (+ 683us) client_negotiation.cc:240] Negotiation successful
0806 10:50:19.605358 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":69,"thread_start_us":1473,"threads_started":1}
I20260806 10:50:19.606473 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.597115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:35679 (local address 127.16.21.2:59909)
0806 10:50:19.598582 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:19.598599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:19.598771 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:19.599149 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:19.599154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:19.599184 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:19.599637 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.599649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.601020 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.601026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.602848 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:19.602860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.604787 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.604794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:19.604981 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.605023 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 10:50:19.605881 (+ 858us) client_negotiation.cc:240] Negotiation successful
0806 10:50:19.606029 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"mutex_wait_us":2,"thread_start_us":797,"threads_started":1}
I20260806 10:50:19.606820 29973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbede0b81e04e988a3e3d5b1e468904" candidate_uuid: "2f14232e035f46078aa184252eab5b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f31ca2385f449d3bbb73543991bcab4" is_pre_election: true
I20260806 10:50:19.607362 29973 raft_consensus.cc:2454] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f14232e035f46078aa184252eab5b9f in term 0.
I20260806 10:50:19.607529 29693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbede0b81e04e988a3e3d5b1e468904" candidate_uuid: "2f14232e035f46078aa184252eab5b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" is_pre_election: true
I20260806 10:50:19.607988 29771 leader_election.cc:302] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f14232e035f46078aa184252eab5b9f, 6f31ca2385f449d3bbb73543991bcab4; no voters:
I20260806 10:50:19.608003 29693 raft_consensus.cc:2454] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f14232e035f46078aa184252eab5b9f in term 0.
I20260806 10:50:19.608376 30053 raft_consensus.cc:2781] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:50:19.608528 30053 raft_consensus.cc:482] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:50:19.608639 30053 raft_consensus.cc:3037] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:19.610970 30053 raft_consensus.cc:504] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.611577 30053 leader_election.cc:288] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [CANDIDATE]: Term 1 election: Requested vote from peers 6f31ca2385f449d3bbb73543991bcab4 (127.16.21.3:39903), 2e344caa892f4f59bc29e9fdbddeaec2 (127.16.21.1:35679)
I20260806 10:50:19.611927 29973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbede0b81e04e988a3e3d5b1e468904" candidate_uuid: "2f14232e035f46078aa184252eab5b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f31ca2385f449d3bbb73543991bcab4"
I20260806 10:50:19.612142 29973 raft_consensus.cc:3037] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:19.612145 29693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbede0b81e04e988a3e3d5b1e468904" candidate_uuid: "2f14232e035f46078aa184252eab5b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e344caa892f4f59bc29e9fdbddeaec2"
I20260806 10:50:19.612360 29693 raft_consensus.cc:3037] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:19.615445 29973 raft_consensus.cc:2454] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f14232e035f46078aa184252eab5b9f in term 1.
I20260806 10:50:19.615506 29693 raft_consensus.cc:2454] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f14232e035f46078aa184252eab5b9f in term 1.
I20260806 10:50:19.615849 29771 leader_election.cc:302] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f14232e035f46078aa184252eab5b9f, 6f31ca2385f449d3bbb73543991bcab4; no voters:
I20260806 10:50:19.616181 30053 raft_consensus.cc:2781] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:19.616742 30053 raft_consensus.cc:686] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 1 LEADER]: Becoming Leader. State: Replica: 2f14232e035f46078aa184252eab5b9f, State: Running, Role: LEADER
I20260806 10:50:19.617251 30053 consensus_queue.cc:237] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } }
I20260806 10:50:19.621968 29540 catalog_manager.cc:5495] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f14232e035f46078aa184252eab5b9f (127.16.21.2). New cstate: current_term: 1 leader_uuid: "2f14232e035f46078aa184252eab5b9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f14232e035f46078aa184252eab5b9f" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260806 10:50:19.665226 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.659147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.16.21.2:37685)
0806 10:50:19.659374 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.659380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.659400 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:50:19.659503 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.659509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.659608 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.659768 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:50:19.660163 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.661331 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.662769 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.663405 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.663532 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.663995 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:50:19.664680 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:50:19.664828 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.664875 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.664954 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:50:19.673295 29693 raft_consensus.cc:1261] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 1 FOLLOWER]: Refusing update from remote peer 2f14232e035f46078aa184252eab5b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:19.673383 29973 raft_consensus.cc:1261] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 1 FOLLOWER]: Refusing update from remote peer 2f14232e035f46078aa184252eab5b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:19.674394 30057 consensus_queue.cc:1035] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f31ca2385f449d3bbb73543991bcab4" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 39903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:19.674798 30053 consensus_queue.cc:1035] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e344caa892f4f59bc29e9fdbddeaec2" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:19.688387 30060 mvcc.cc:205] Tried to move back new op lower bound from 7315510966972108800 to 7315510966756311040. Current Snapshot: MvccSnapshot[applied={T|T < 7315510966972108800}]
I20260806 10:50:19.690056 30061 mvcc.cc:205] Tried to move back new op lower bound from 7315510966972108800 to 7315510966756311040. Current Snapshot: MvccSnapshot[applied={T|T < 7315510966972108800}]
I20260806 10:50:19.713989 29540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:19.714309 29540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:19.718001 29540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df201571b8ed4f5a85dea198574d2161: Sending DeleteTablet for 3 replicas of tablet afbede0b81e04e988a3e3d5b1e468904
I20260806 10:50:19.719173 29953 tablet_service.cc:1514] Processing DeleteTablet for tablet afbede0b81e04e988a3e3d5b1e468904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:33840
I20260806 10:50:19.719187 29818 tablet_service.cc:1514] Processing DeleteTablet for tablet afbede0b81e04e988a3e3d5b1e468904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:56904
I20260806 10:50:19.720121 29673 tablet_service.cc:1514] Processing DeleteTablet for tablet afbede0b81e04e988a3e3d5b1e468904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:45856
I20260806 10:50:19.728515 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29577
I20260806 10:50:19.732750 30072 tablet_replica.cc:330] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: stopping tablet replica
I20260806 10:50:19.733450 30070 tablet_replica.cc:330] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: stopping tablet replica
I20260806 10:50:19.734619 30071 tablet_replica.cc:330] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: stopping tablet replica
I20260806 10:50:19.734829 30072 raft_consensus.cc:2229] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:19.735357 30072 raft_consensus.cc:2258] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:19.735939 30070 raft_consensus.cc:2229] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:19.736409 30070 raft_consensus.cc:2258] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:19.738253 30072 ts_tablet_manager.cc:1905] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:19.739374 30070 ts_tablet_manager.cc:1905] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:19.745457 30072 ts_tablet_manager.cc:1918] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:19.745616 30072 log.cc:1196] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/afbede0b81e04e988a3e3d5b1e468904
I20260806 10:50:19.746212 30072 ts_tablet_manager.cc:1939] T afbede0b81e04e988a3e3d5b1e468904 P 2e344caa892f4f59bc29e9fdbddeaec2: Deleting consensus metadata
I20260806 10:50:19.746482 30070 ts_tablet_manager.cc:1918] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:19.746645 30070 log.cc:1196] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/afbede0b81e04e988a3e3d5b1e468904
I20260806 10:50:19.747190 30070 ts_tablet_manager.cc:1939] T afbede0b81e04e988a3e3d5b1e468904 P 6f31ca2385f449d3bbb73543991bcab4: Deleting consensus metadata
I20260806 10:50:19.747998 30071 raft_consensus.cc:2229] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:50:19.747968 29525 catalog_manager.cc:4841] TS 2e344caa892f4f59bc29e9fdbddeaec2 (127.16.21.1:35679): tablet afbede0b81e04e988a3e3d5b1e468904 (table test-table [id=208292c31209490ebef8efe2a8fa25ab]) successfully deleted
I20260806 10:50:19.748692 30071 raft_consensus.cc:2258] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:19.748808 29524 catalog_manager.cc:4841] TS 6f31ca2385f449d3bbb73543991bcab4 (127.16.21.3:39903): tablet afbede0b81e04e988a3e3d5b1e468904 (table test-table [id=208292c31209490ebef8efe2a8fa25ab]) successfully deleted
I20260806 10:50:19.750049 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.730088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.16.21.1:35679)
0806 10:50:19.730269 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.730275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.730294 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:50:19.730467 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.730475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.730583 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.730677 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:19.733760 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.734936 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.736577 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.737244 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.737401 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.740782 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:19.740795 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:19.740810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:19.740858 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:19.744884 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.745898 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.745906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.745910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.746071 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.746790 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.746797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.746801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.747376 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:19.747694 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.749406 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.749633 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:50:19.750422 30071 ts_tablet_manager.cc:1905] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:19.750880 29733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:19.758252 30071 ts_tablet_manager.cc:1918] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:19.758423 30071 log.cc:1196] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/afbede0b81e04e988a3e3d5b1e468904
I20260806 10:50:19.758987 30071 ts_tablet_manager.cc:1939] T afbede0b81e04e988a3e3d5b1e468904 P 2f14232e035f46078aa184252eab5b9f: Deleting consensus metadata
I20260806 10:50:19.760650 29527 catalog_manager.cc:4841] TS 2f14232e035f46078aa184252eab5b9f (127.16.21.2:37685): tablet afbede0b81e04e988a3e3d5b1e468904 (table test-table [id=208292c31209490ebef8efe2a8fa25ab]) successfully deleted
I20260806 10:50:19.861649 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29577
I20260806 10:50:19.887876 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29743
I20260806 10:50:19.908048 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:19.888743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.16.21.2:37685)
0806 10:50:19.888939 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:19.888946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:19.888968 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:50:19.889121 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:19.889126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:19.889218 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:19.889326 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:19.890582 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.891854 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.893311 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:19.894010 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:19.894130 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:19.901338 (+ 7208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:19.901352 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:19.901377 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:19.901423 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:19.905319 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.905948 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.905958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.905962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.906151 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:19.906516 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:19.906523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:19.906526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:19.906980 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:19.907292 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:19.907439 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 10:50:19.907670 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 10:50:19.908829 29879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:20.029172 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29743
I20260806 10:50:20.050390 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29888
I20260806 10:50:20.064028 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:20.051181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.16.21.3:39903)
0806 10:50:20.051374 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:20.051381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:20.051431 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:50:20.051659 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:20.051664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:20.051728 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:20.051852 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:20.053280 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:20.054003 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:20.055307 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:20.055703 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:20.055840 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:20.058414 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:20.058427 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:20.058444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:20.058475 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:20.061185 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:20.061902 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:20.061910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:20.061912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:20.062039 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:20.062463 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:20.062469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:20.062472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:20.062833 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:20.063155 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:20.063432 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 10:50:20.063644 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:50:20.064751 30013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:20.154686 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29888
I20260806 10:50:20.172564 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 29507
I20260806 10:50:20.174088 29570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:20.288808 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 29507
I20260806 10:50:20.318797 16468 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 10:50:20.663990 16468 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 10:50:20.664297 16468 mini_ranger.cc:66] Stopping Ranger...
I20260806 10:50:21.009305 16468 mini_ranger.cc:68] Stopped Ranger
2026-08-06 10:50:21.009 UTC [28552] LOG: received smart shutdown request
2026-08-06 10:50:21.012 UTC [28552] LOG: background worker "logical replication launcher" (PID 28561) exited with exit code 1
2026-08-06 10:50:21.012 UTC [28556] LOG: shutting down
2026-08-06 10:50:21.167 UTC [28552] LOG: database system is shut down
2026-08-06T10:50:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51753 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:50:21 dist-test-slave-0b58 krb5kdc[30085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:50:21 dist-test-slave-0b58 krb5kdc[30085](info): set up 2 sockets
Aug 06 10:50:21 dist-test-slave-0b58 krb5kdc[30085](info): commencing operation
krb5kdc: starting...
W20260806 10:50:22.428433 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:50:22 dist-test-slave-0b58 krb5kdc[30085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:50:22.691632 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:34305
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:34305,127.16.21.61:35287,127.16.21.60:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:23.025446 30098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:23.025707 30098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:23.025878 30098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:23.032927 30098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:23.033041 30098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:23.033121 30098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:23.033226 30098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:23.033289 30098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:23.042838 30098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:34305,127.16.21.61:35287,127.16.21.60:41417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:23.044054 30098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:23.045334 30098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:23.057044 30104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:23.057129 30106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:23.057164 30103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:23.057873 30098 server_base.cc:1034] running on GCE node
I20260806 10:50:23.058521 30098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:23.059708 30098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:23.060920 30098 hybrid_clock.cc:648] HybridClock initialized: now 1786013423060867 us; error 48 us; skew 500 ppm
I20260806 10:50:23.061272 30098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:23 dist-test-slave-0b58 krb5kdc[30085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:23.078764 30098 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:23.080142 30098 webserver.cc:460] Webserver started at http://127.16.21.62:45147/ using document root <none> and password file <none>
I20260806 10:50:23.080874 30098 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:23.081027 30098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:23.081338 30098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:23.084335 30098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "727b1210673445809a08a54b4059b466"
format_stamp: "Formatted at 2026-08-06 10:50:23 on dist-test-slave-0b58"
server_key: "40211f01991be13f9a2081878361d259"
server_key_iv: "bb3c8e358427bd17516147f6bde36d7f"
server_key_version: "encryptionkey@0"
I20260806 10:50:23.085187 30098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "727b1210673445809a08a54b4059b466"
format_stamp: "Formatted at 2026-08-06 10:50:23 on dist-test-slave-0b58"
server_key: "40211f01991be13f9a2081878361d259"
server_key_iv: "bb3c8e358427bd17516147f6bde36d7f"
server_key_version: "encryptionkey@0"
I20260806 10:50:23.090092 30098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:50:23.093271 30114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:23.094472 30098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:50:23.094650 30098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "727b1210673445809a08a54b4059b466"
format_stamp: "Formatted at 2026-08-06 10:50:23 on dist-test-slave-0b58"
server_key: "40211f01991be13f9a2081878361d259"
server_key_iv: "bb3c8e358427bd17516147f6bde36d7f"
server_key_version: "encryptionkey@0"
I20260806 10:50:23.094858 30098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:23.128940 30098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:23.132748 30098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:23.133026 30098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:23.151885 30098 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:34305
I20260806 10:50:23.151934 30165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:34305 every 8 connection(s)
I20260806 10:50:23.153344 30098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:23.155586 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30098
I20260806 10:50:23.155844 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:50:23.156728 16468 external_mini_cluster.cc:1428] Setting key 6a0b352bb331cb15b00aabada94bf873
I20260806 10:50:23.157675 30166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:23.167680 30166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:50:23 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.61@KRBTEST.COM" created.
I20260806 10:50:23.191634 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.171704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34305 (local address 127.0.0.1:54322)
0806 10:50:23.172572 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.172630 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:23.175502 (+ 2872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:23.176437 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:23.176443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:23.177156 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:23.177481 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.177498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.179047 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.179054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.180118 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.180130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.181056 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.181063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.181324 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.182028 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:23.182049 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:23.184173 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:23.187629 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.187645 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.187658 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.188288 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.188736 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.188741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.188745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.188923 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.189719 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:23.189725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:23.189896 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 10:50:23.190082 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 10:50:23.190484 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1,"thread_start_us":299,"threads_started":1}
I20260806 10:50:23.191651 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.172862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.16.21.62:34305)
0806 10:50:23.173490 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:23.173496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:23.173535 (+ 39us) server_negotiation.cc:407] Connection header received
0806 10:50:23.175881 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:23.175886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:23.176127 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:23.176416 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:23.177680 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.178894 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.180296 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.180826 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.181027 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.184443 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:23.184456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:23.184475 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:23.184515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:23.187489 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.188403 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.188409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.188411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.188618 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.189026 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.189029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.189031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.189560 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:23.189727 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:23.190219 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 10:50:23.190645 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":479,"thread_start_us":303,"threads_started":1}
I20260806 10:50:23.194789 30166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:50:23.195775 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.195398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60562)
0806 10:50:23.195577 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.195679 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 10:50:23.196084 30116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
I20260806 10:50:23.197630 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.197294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60564)
0806 10:50:23.197457 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.197521 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 10:50:23.197897 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35287: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab.
Entry for principal kudu/127.16.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab.
I20260806 10:50:23.225268 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:35287
--webserver_interface=127.16.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab
--principal=kudu/127.16.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:34305,127.16.21.61:35287,127.16.21.60:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:23.247304 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } attempt: 1
I20260806 10:50:23.248550 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.248054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60572)
0806 10:50:23.248306 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.248410 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 10:50:23.249565 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.249249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60580)
0806 10:50:23.249381 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.249443 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 10:50:23.249795 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35287: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
I20260806 10:50:23.318140 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } attempt: 2
I20260806 10:50:23.319413 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.318936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60596)
0806 10:50:23.319170 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.319260 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 10:50:23.320508 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.320112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60608)
0806 10:50:23.320243 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.320386 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 10:50:23.320809 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35287: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
I20260806 10:50:23.412163 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } attempt: 3
I20260806 10:50:23.413398 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.412951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60610)
0806 10:50:23.413150 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.413252 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 10:50:23.414422 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.414094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60624)
0806 10:50:23.414229 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.414297 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 10:50:23.414698 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35287: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
I20260806 10:50:23.557948 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } attempt: 4
I20260806 10:50:23.559099 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.558595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60638)
0806 10:50:23.558895 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.558977 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260806 10:50:23.559435 30173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:23.559791 30173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:23.559947 30173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 10:50:23.560238 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.559860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60650)
0806 10:50:23.560019 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.560124 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:50:23.560473 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35287: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35287: connect: Connection refused (error 111)
W20260806 10:50:23.567157 30173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:23.567278 30173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:23.567377 30173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:23.567477 30173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:23.567540 30173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:23.577199 30173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:34305,127.16.21.61:35287,127.16.21.60:41417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab
--principal=kudu/127.16.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:35287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:23.578486 30173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:23.579852 30173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:23.591297 30178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:23.591346 30181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:23.591370 30179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:23.592211 30173 server_base.cc:1034] running on GCE node
I20260806 10:50:23.592971 30173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:23.594214 30173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:23.595449 30173 hybrid_clock.cc:648] HybridClock initialized: now 1786013423595414 us; error 44 us; skew 500 ppm
I20260806 10:50:23.595781 30173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:23 dist-test-slave-0b58 krb5kdc[30085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:23.598903 30173 init.cc:377] Logged in from keytab as kudu/127.16.21.61@KRBTEST.COM (short username kudu)
I20260806 10:50:23.600332 30173 webserver.cc:460] Webserver started at http://127.16.21.61:38675/ using document root <none> and password file <none>
I20260806 10:50:23.600965 30173 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:23.601109 30173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:23.601451 30173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:23.604328 30173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data/instance:
uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a"
format_stamp: "Formatted at 2026-08-06 10:50:23 on dist-test-slave-0b58"
server_key: "baffcfb6ad49f623d5836db98393453b"
server_key_iv: "c67a0a7622dc2ba072866c3dd9c4a9eb"
server_key_version: "encryptionkey@0"
I20260806 10:50:23.605151 30173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal/instance:
uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a"
format_stamp: "Formatted at 2026-08-06 10:50:23 on dist-test-slave-0b58"
server_key: "baffcfb6ad49f623d5836db98393453b"
server_key_iv: "c67a0a7622dc2ba072866c3dd9c4a9eb"
server_key_version: "encryptionkey@0"
I20260806 10:50:23.610363 30173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260806 10:50:23.613621 30188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:23.614858 30173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 10:50:23.615031 30173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal
uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a"
format_stamp: "Formatted at 2026-08-06 10:50:23 on dist-test-slave-0b58"
server_key: "baffcfb6ad49f623d5836db98393453b"
server_key_iv: "c67a0a7622dc2ba072866c3dd9c4a9eb"
server_key_version: "encryptionkey@0"
I20260806 10:50:23.615223 30173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:23.641469 30173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:23.645328 30173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:23.645620 30173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:23.664383 30173 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.61:35287
I20260806 10:50:23.664469 30239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.61:35287 every 8 connection(s)
I20260806 10:50:23.665781 30173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
I20260806 10:50:23.669436 30240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:23.671284 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30173
I20260806 10:50:23.671489 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-1/wal/instance
I20260806 10:50:23.672050 16468 external_mini_cluster.cc:1428] Setting key 90d5e59c8763dc09ffa94793a9b96f11
I20260806 10:50:23.678928 30240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:50:23 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:23.701664 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.682311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.16.21.62:34305)
0806 10:50:23.682549 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:23.682557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:23.683692 (+ 1135us) server_negotiation.cc:407] Connection header received
0806 10:50:23.685599 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:23.685605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:23.685691 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:23.685809 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:23.687819 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.688572 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.691060 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.691473 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.691565 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.695696 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:23.695701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:23.695704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:23.695737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:23.698547 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.699356 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.699364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.699367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.699491 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.699945 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.699951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.699955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.700232 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:23.700461 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:23.701086 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 10:50:23.701324 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
WARNING: no policy specified for kudu/127.16.21.60@KRBTEST.COM; defaulting to no policy
I20260806 10:50:23.702576 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.682680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34305 (local address 127.0.0.1:54334)
0806 10:50:23.683503 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.683554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:23.685334 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:23.686039 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:23.686046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:23.686819 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:23.687574 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.687601 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.688745 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.688753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.690916 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.690934 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.691695 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.691704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.692013 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.692773 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:23.692811 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:23.695465 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:23.698686 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.698694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.698708 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.699236 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.699605 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.699610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.699614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.699826 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.700415 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:23.700423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:23.700734 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 10:50:23.700998 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 10:50:23.701465 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":362,"threads_started":1}
I20260806 10:50:23.704177 30240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.60@KRBTEST.COM" created.
Aug 06 10:50:23 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:50:23.719031 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.704923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60656)
0806 10:50:23.705108 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.705126 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:23.705199 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:23.706217 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:23.706222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:23.706642 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:23.706764 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.706771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.708073 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.708078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.708957 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.708968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.709731 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.709736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.709954 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.710483 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:23.710498 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:23.712905 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:23.716560 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.716563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.716566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.716914 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.717309 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.717313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.717315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.717449 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.718200 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:23.718205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:23.718340 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 10:50:23.718449 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 10:50:23.718688 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 10:50:23.719468 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.705234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.16.21.61:35287)
0806 10:50:23.705853 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:23.705858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:23.705882 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:50:23.705920 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:23.705925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:23.706066 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:23.706228 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:23.706910 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.707936 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.709090 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.709593 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.709725 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.713107 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:23.713117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:23.713127 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:23.713155 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:23.716424 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.717038 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.717049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.717054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.717199 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.717552 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.717560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.717566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.718040 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:23.718291 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:23.718643 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 10:50:23.718943 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":484,"thread_start_us":351,"threads_started":1}
I20260806 10:50:23.721536 30240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:50:23.722536 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.722132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51596)
0806 10:50:23.722296 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.722397 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 10:50:23.722755 30189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:23.724241 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.723930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51602)
0806 10:50:23.724080 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.724153 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 10:50:23.724570 30240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab.
Entry for principal kudu/127.16.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab.
I20260806 10:50:23.739133 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:41417
--webserver_interface=127.16.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab
--principal=kudu/127.16.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:34305,127.16.21.61:35287,127.16.21.60:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:23.773840 30240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 1
I20260806 10:50:23.774883 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.774441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51606)
0806 10:50:23.774632 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.774733 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 10:50:23.775856 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.775509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51618)
0806 10:50:23.775666 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.775742 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 10:50:23.776057 30240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:23.844275 30240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 2
I20260806 10:50:23.845329 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.844883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51622)
0806 10:50:23.845079 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.845176 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 10:50:23.846357 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.846021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51626)
0806 10:50:23.846184 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.846247 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 10:50:23.846557 30240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:23.859767 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } attempt: 5
Aug 06 10:50:23 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:50:23.874243 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.860388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60658)
0806 10:50:23.860642 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.860661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:23.860792 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:23.861470 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:23.861477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:23.861991 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:23.862162 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.862170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.863532 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.863535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.864469 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:23.864481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.865399 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.865403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:23.865530 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.866155 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:23.866184 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:23.868239 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:23.872018 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.872022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.872024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.872438 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.872854 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:23.872857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:23.872859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:23.872984 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:23.873578 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:23.873581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:23.873705 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 10:50:23.873812 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:50:23.874000 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260806 10:50:23.874707 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.860556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.16.21.61:35287)
0806 10:50:23.861026 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:23.861032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:23.861076 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:50:23.861173 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:23.861178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:23.861252 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:23.861357 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:23.862328 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.863398 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.864599 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:23.865245 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:23.865380 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:23.868483 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:23.868490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:23.868495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:23.868532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:23.871871 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.872573 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.872581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.872586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.872749 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:23.873095 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:23.873100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:23.873103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:23.873468 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:23.873614 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:23.873991 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 10:50:23.874255 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260806 10:50:23.875682 30166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:50:23.876822 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.876456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51628)
0806 10:50:23.876606 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.876677 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 10:50:23.877868 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.877515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51632)
0806 10:50:23.877703 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.877772 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260806 10:50:23.878080 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:23.929322 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 1
I20260806 10:50:23.930492 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.929991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51642)
0806 10:50:23.930203 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.930328 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 10:50:23.931491 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.931153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51656)
0806 10:50:23.931320 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.931384 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 10:50:23.931704 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:23.937744 30240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 3
I20260806 10:50:23.938804 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.938333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51666)
0806 10:50:23.938580 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.938663 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 10:50:23.939821 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:23.939405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51672)
0806 10:50:23.939640 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:23.939710 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260806 10:50:23.940032 30240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:23.999892 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 2
I20260806 10:50:24.001060 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.000583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51686)
0806 10:50:24.000802 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.000898 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 10:50:24.002019 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.001682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51696)
0806 10:50:24.001830 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.001903 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 10:50:24.002303 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
W20260806 10:50:24.071321 30247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:24.071633 30247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:24.071803 30247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:24.078773 30247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:24.078943 30247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:24.079046 30247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:24.079149 30247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:24.079233 30247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:24.083227 30240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 4
I20260806 10:50:24.084370 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.083843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51710)
0806 10:50:24.084081 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.084166 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 10:50:24.085366 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.085051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51714)
0806 10:50:24.085204 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.085265 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 10:50:24.085565 30240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
I20260806 10:50:24.089171 30247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:34305,127.16.21.61:35287,127.16.21.60:41417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab
--principal=kudu/127.16.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:24.090390 30247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:24.091617 30247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:24.103451 30252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:24.103509 30253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:24.103415 30255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:24.104559 30247 server_base.cc:1034] running on GCE node
I20260806 10:50:24.105312 30247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:24.106523 30247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:24.107753 30247 hybrid_clock.cc:648] HybridClock initialized: now 1786013424107716 us; error 46 us; skew 500 ppm
I20260806 10:50:24.108074 30247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:50:24.108496 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 3
I20260806 10:50:24.109462 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.109028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51726)
0806 10:50:24.109239 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.109323 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 10:50:24.110452 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.110073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51742)
0806 10:50:24.110275 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.110349 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260806 10:50:24.110678 30166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:41417: Network error: Client connection negotiation failed: client connection to 127.16.21.60:41417: connect: Connection refused (error 111)
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:24.111589 30247 init.cc:377] Logged in from keytab as kudu/127.16.21.60@KRBTEST.COM (short username kudu)
I20260806 10:50:24.113109 30247 webserver.cc:460] Webserver started at http://127.16.21.60:42671/ using document root <none> and password file <none>
I20260806 10:50:24.113749 30247 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:24.113888 30247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:24.114171 30247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:24.117004 30247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data/instance:
uuid: "b0ab7d5062b0461ab3f724c1295dbc0f"
format_stamp: "Formatted at 2026-08-06 10:50:24 on dist-test-slave-0b58"
server_key: "c61ab51af58e44f8e072c3e70a5c9231"
server_key_iv: "d607b526ce8f7cdd55e1298ab1236464"
server_key_version: "encryptionkey@0"
I20260806 10:50:24.117777 30247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal/instance:
uuid: "b0ab7d5062b0461ab3f724c1295dbc0f"
format_stamp: "Formatted at 2026-08-06 10:50:24 on dist-test-slave-0b58"
server_key: "c61ab51af58e44f8e072c3e70a5c9231"
server_key_iv: "d607b526ce8f7cdd55e1298ab1236464"
server_key_version: "encryptionkey@0"
I20260806 10:50:24.122535 30247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:50:24.125849 30262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:24.127028 30247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 10:50:24.127203 30247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal
uuid: "b0ab7d5062b0461ab3f724c1295dbc0f"
format_stamp: "Formatted at 2026-08-06 10:50:24 on dist-test-slave-0b58"
server_key: "c61ab51af58e44f8e072c3e70a5c9231"
server_key_iv: "d607b526ce8f7cdd55e1298ab1236464"
server_key_version: "encryptionkey@0"
I20260806 10:50:24.127422 30247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:24.143265 30247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:24.147399 30247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:24.147666 30247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:24.166730 30247 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.60:41417
I20260806 10:50:24.166771 30315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.60:41417 every 8 connection(s)
I20260806 10:50:24.167999 30247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
I20260806 10:50:24.171397 30316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:24.172078 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30247
I20260806 10:50:24.172323 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013221924036-16468-0/minicluster-data/master-2/wal/instance
I20260806 10:50:24.172925 16468 external_mini_cluster.cc:1428] Setting key ec309f30dfa46ed2ca58e9cd2076b81b
I20260806 10:50:24.180814 30316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:24.197256 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.174853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.16.21.60:41417)
0806 10:50:24.175669 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.175679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.175727 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:50:24.177671 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.177691 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.178057 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.178366 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.180163 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.181585 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.183540 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.184631 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.184826 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.189767 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.189779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.189789 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.189848 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.192873 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.193665 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.193680 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.193685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.193826 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.194263 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.194269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.194272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.194932 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.195212 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.195602 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.195986 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":424,"threads_started":1}
I20260806 10:50:24.198520 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.183469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.16.21.62:34305)
0806 10:50:24.183690 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.183698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.184455 (+ 757us) server_negotiation.cc:407] Connection header received
0806 10:50:24.184622 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.184626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.184683 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.184799 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.186011 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.187031 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.188826 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.189259 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.189400 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.192639 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.192647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.192656 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.192687 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.195711 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.196494 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.196499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.196503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.196677 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.197075 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.197079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.197082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.197424 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.197605 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.198106 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.198261 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:50:24.198725 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.183450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:34305 (local address 127.0.0.1:54340)
0806 10:50:24.184309 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.184361 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:24.184515 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:24.184867 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:24.184873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:24.185578 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:24.185866 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.185875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.187215 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.187222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.188674 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.188687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.189486 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.189495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.189664 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.190325 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:24.190342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:24.192458 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:24.195860 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.195876 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.195888 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.196358 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.196792 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.196795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.196797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.196943 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.197579 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:24.197588 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:24.197801 (+ 213us) client_negotiation.cc:769] Sending connection context
0806 10:50:24.198004 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 10:50:24.198278 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":418,"threads_started":1}
I20260806 10:50:24.200043 30316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:24.214471 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.200646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35287 (local address 127.0.0.1:60674)
0806 10:50:24.200819 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.200836 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:24.200920 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:24.201514 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:24.201521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:24.201984 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:24.202177 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.202191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.203586 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.203590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.204918 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.204933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.205737 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.205741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.205911 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.206458 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:24.206483 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:24.208726 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:24.212129 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.212136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.212141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.212673 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.213041 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.213046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.213049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.213234 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.213700 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:24.213706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:24.213882 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 10:50:24.214031 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 10:50:24.214247 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:50:24.214812 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.201021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.16.21.61:35287)
0806 10:50:24.201204 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.201210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.201228 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:50:24.201284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.201289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.201357 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.201443 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.202320 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.203412 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.205060 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.205603 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.205723 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.208931 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.208940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.208947 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.208991 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.212001 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.212792 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.212796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.212798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.212923 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.213348 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.213351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.213353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.213583 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.213693 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.214197 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.214443 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 10:50:24.215710 30316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:50:24.216856 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.200543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.16.21.62:34305)
0806 10:50:24.200766 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.200774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.200797 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:50:24.201003 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.201008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.201069 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.201157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.202636 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.203671 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.205380 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.205965 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.206050 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.210963 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.210971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.210978 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.211015 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.213961 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.214725 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.214730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.214733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.214902 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.215354 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.215359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.215362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.215714 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.215961 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.216338 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.216552 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:50:24.230803 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.216494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.16.21.60:41417)
0806 10:50:24.216651 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.216657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.216870 (+ 213us) server_negotiation.cc:407] Connection header received
0806 10:50:24.217026 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.217033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.217092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.217153 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.218108 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.219175 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.220322 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.220740 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.220868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.224069 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.224074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.224077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.224124 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.227679 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.228701 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.228707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.228710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.228880 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.229239 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.229244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.229247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.229613 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.229762 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.230096 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.230464 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":53}
I20260806 10:50:24.231137 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.216536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51774)
0806 10:50:24.216752 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.216779 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:24.216906 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:24.217328 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:24.217332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:24.217796 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:24.217972 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.217983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.219287 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.219290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.220180 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.220191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.220876 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.220881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.221113 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.221673 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:24.221692 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:24.223880 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:24.227892 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.227897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.227901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.228556 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.228998 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.229002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.229005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.229131 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.229730 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:24.229734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:24.229878 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 10:50:24.230011 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 10:50:24.230300 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2}
I20260806 10:50:24.235447 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.218805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.16.21.61:35287)
0806 10:50:24.219003 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.219010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.219030 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:50:24.219081 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.219086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.219150 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.219244 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.220581 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.221313 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.223221 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.224298 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.224436 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.229837 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.229843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.229847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.229884 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.232899 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.233730 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.233734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.233736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.233868 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.234265 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.234268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.234270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.234505 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.234721 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.235053 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.235215 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:50:24.237098 30316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: Bootstrap starting.
I20260806 10:50:24.240355 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:50:24.240921 30316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:24.241914 30316 log.cc:824] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:24.244678 30316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: No bootstrap required, opened a new log
I20260806 10:50:24.257351 30316 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.257875 30316 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:24.258037 30316 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ab7d5062b0461ab3f724c1295dbc0f, State: Initialized, Role: FOLLOWER
I20260806 10:50:24.258860 30316 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.264967 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.245259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.16.21.62:34305)
0806 10:50:24.245487 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.245493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.245847 (+ 354us) server_negotiation.cc:407] Connection header received
0806 10:50:24.246177 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.246183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.246271 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.246346 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.248867 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.249954 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.252312 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.253039 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.253148 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.257399 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.257408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.257416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.257476 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.261352 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.262330 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.262338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.262343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.262523 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.263003 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.263008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.263012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.263372 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.263613 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.264319 (+ 706us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.264582 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 10:50:24.269814 30316 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:24.270481 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.271008 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.273200 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.246403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.16.21.61:35287)
0806 10:50:24.251602 (+ 5199us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.251610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.256809 (+ 5199us) server_negotiation.cc:407] Connection header received
0806 10:50:24.257140 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.257146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.257221 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.257305 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.259373 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.260596 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.262696 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.263370 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.263473 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.266959 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.266965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.266970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.267013 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.269782 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.270892 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.270898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.270901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.271068 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.271477 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.271488 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.271492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.271919 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.272172 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.272667 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.272894 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5031}
I20260806 10:50:24.277572 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.246148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.16.21.60:41417)
0806 10:50:24.247147 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.247156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.257222 (+ 10066us) server_negotiation.cc:407] Connection header received
0806 10:50:24.257505 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.257509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.257585 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.257659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.259781 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.261266 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.264152 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.265172 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.265307 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.269310 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.269316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.269320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.269357 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.274251 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.275282 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.275288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.275295 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.275509 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.275983 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.275986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.275989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.276376 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.276687 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.276996 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.277218 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693}
W20260806 10:50:24.284806 30340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:50:24.284943 30340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:50:24.287887 30166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 4
I20260806 10:50:24.281356 30247 master_runner.cc:186] Error getting master registration for 127.16.21.62:34305: OK, instance_id { permanent_uuid: "727b1210673445809a08a54b4059b466" instance_seqno: 1786013423146469 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:50:24.293439 30247 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
I20260806 10:50:24.304530 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.288543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51790)
0806 10:50:24.288857 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.288874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:24.288975 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:24.289498 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:24.289503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:24.289987 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:24.290201 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.290213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.291777 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.291784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.293289 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.293305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.294510 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.294516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.294739 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.295307 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:24.295331 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:24.298113 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:24.302072 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.302079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.302083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.302459 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.302890 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.302896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.302900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.303026 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.303535 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:24.303540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:24.303679 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 10:50:24.303819 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 10:50:24.304100 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260806 10:50:24.304543 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.288882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.16.21.60:41417)
0806 10:50:24.289041 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.289047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.289065 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:50:24.289111 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.289116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.289206 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.289324 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.290365 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.291637 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.293478 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.294156 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.294273 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.298296 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.298303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.298310 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.298349 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.301949 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.302566 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.302572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.302576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.302776 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.303140 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.303145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.303148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.303408 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.303552 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.303907 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.304121 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 10:50:24.308997 30166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: Bootstrap starting.
I20260806 10:50:24.312057 30166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:24.313056 30166 log.cc:824] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:24.315491 30166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: No bootstrap required, opened a new log
I20260806 10:50:24.323132 30166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.323362 30166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:24.323505 30166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727b1210673445809a08a54b4059b466, State: Initialized, Role: FOLLOWER
I20260806 10:50:24.324043 30166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.324905 30343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.325270 30343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.325948 30166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:50:24.336719 30354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:50:24.336880 30354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:50:24.335645 30098 master_runner.cc:186] Error getting master registration for 127.16.21.61:35287: OK, instance_id { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" instance_seqno: 1786013423659035 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:50:24.345000 30098 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:50:24.384756 30240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } attempt: 5
W20260806 10:50:24.385665 30120 tablet.cc:2369] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 10:50:24 dist-test-slave-0b58 krb5kdc[30085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
W20260806 10:50:24.399813 30269 tablet.cc:2369] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:24.402374 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.385370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:41417 (local address 127.0.0.1:51800)
0806 10:50:24.385597 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:24.385617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:24.385757 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:24.386104 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:24.386108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:24.386718 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:24.386945 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.386958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.388505 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.388511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.390043 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:24.390061 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.391092 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.391098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:24.391325 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.392024 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:24.392048 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:24.394848 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:24.399250 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.399255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.399259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.399821 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.400406 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:24.400430 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:24.400433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:24.400600 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:24.401343 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:24.401349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:24.401510 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 10:50:24.401665 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 10:50:24.401954 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 10:50:24.403481 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:24.385397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.16.21.60:41417)
0806 10:50:24.385566 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:24.385572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:24.385702 (+ 130us) server_negotiation.cc:407] Connection header received
0806 10:50:24.385902 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:24.385907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:24.385973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:24.386078 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:24.387111 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.388348 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.390198 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:24.390931 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:24.391056 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:24.395051 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:24.395057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:24.395063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:24.395100 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:24.399077 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.400051 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.400057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.400060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.400224 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:24.400737 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:24.400742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:24.400746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:24.401108 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:24.401263 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:24.402863 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0806 10:50:24.403114 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 10:50:24.409343 30240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a: Bootstrap starting.
I20260806 10:50:24.413239 30240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:24.414487 30240 log.cc:824] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:24.417548 30240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a: No bootstrap required, opened a new log
I20260806 10:50:24.429991 30240 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.430379 30240 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:24.430500 30240 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a8e608c1284ef3bdd6ac96739d0b7a, State: Initialized, Role: FOLLOWER
I20260806 10:50:24.431205 30240 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.432369 30357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.432848 30357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.433518 30240 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:24.445370 30357 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:50:24.445631 30357 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.447140 30357 leader_election.cc:288] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 727b1210673445809a08a54b4059b466 (127.16.21.62:34305), b0ab7d5062b0461ab3f724c1295dbc0f (127.16.21.60:41417)
I20260806 10:50:24.448551 30141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727b1210673445809a08a54b4059b466" is_pre_election: true
I20260806 10:50:24.448588 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" is_pre_election: true
I20260806 10:50:24.449139 30141 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a8e608c1284ef3bdd6ac96739d0b7a in term 0.
I20260806 10:50:24.449168 30290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a8e608c1284ef3bdd6ac96739d0b7a in term 0.
I20260806 10:50:24.449730 30190 leader_election.cc:302] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 727b1210673445809a08a54b4059b466, d4a8e608c1284ef3bdd6ac96739d0b7a; no voters:
I20260806 10:50:24.450114 30357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:50:24.450260 30357 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:50:24.450464 30357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:24.453709 30357 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.454375 30357 leader_election.cc:288] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [CANDIDATE]: Term 1 election: Requested vote from peers 727b1210673445809a08a54b4059b466 (127.16.21.62:34305), b0ab7d5062b0461ab3f724c1295dbc0f (127.16.21.60:41417)
I20260806 10:50:24.455331 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f"
I20260806 10:50:24.455540 30290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:24.455928 30141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727b1210673445809a08a54b4059b466"
I20260806 10:50:24.456111 30141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:24.458823 30290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a8e608c1284ef3bdd6ac96739d0b7a in term 1.
I20260806 10:50:24.459148 30189 leader_election.cc:302] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ab7d5062b0461ab3f724c1295dbc0f, d4a8e608c1284ef3bdd6ac96739d0b7a; no voters:
I20260806 10:50:24.459468 30357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:24.459801 30141 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a8e608c1284ef3bdd6ac96739d0b7a in term 1.
I20260806 10:50:24.460712 30357 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [term 1 LEADER]: Becoming Leader. State: Replica: d4a8e608c1284ef3bdd6ac96739d0b7a, State: Running, Role: LEADER
I20260806 10:50:24.461252 30357 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } }
I20260806 10:50:24.465137 30368 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a8e608c1284ef3bdd6ac96739d0b7a. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.465567 30368 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:24.470726 30371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:24.483801 30371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:24.492494 30290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [term 1 FOLLOWER]: Refusing update from remote peer d4a8e608c1284ef3bdd6ac96739d0b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:24.493451 30368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:24.501430 30141 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [term 1 FOLLOWER]: Refusing update from remote peer d4a8e608c1284ef3bdd6ac96739d0b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:24.502372 30368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:24.535036 30343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a8e608c1284ef3bdd6ac96739d0b7a. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.535403 30343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.537374 30373 mvcc.cc:205] Tried to move back new op lower bound from 7315510986710499328 to 7315510986597670912. Current Snapshot: MvccSnapshot[applied={T|T < 7315510986710499328}]
I20260806 10:50:24.539315 30375 mvcc.cc:205] Tried to move back new op lower bound from 7315510986710499328 to 7315510986597670912. Current Snapshot: MvccSnapshot[applied={T|T < 7315510986710499328}]
I20260806 10:50:24.548005 30357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.548422 30357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:24.548801 30343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.549036 30343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.549196 30371 catalog_manager.cc:1347] Generated new cluster ID: 3afc7494d2ca40d984ec3a7da28e27ff
I20260806 10:50:24.549316 30371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:24.553915 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a8e608c1284ef3bdd6ac96739d0b7a. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.554255 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.556844 30368 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.557093 30368 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:24.566519 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727b1210673445809a08a54b4059b466" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 34305 } } peers { permanent_uuid: "d4a8e608c1284ef3bdd6ac96739d0b7a" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35287 } } peers { permanent_uuid: "b0ab7d5062b0461ab3f724c1295dbc0f" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 41417 } } }
I20260806 10:50:24.566742 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:50:24.589370 30371 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:50:24.591257 30371 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:24.609568 30371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a8e608c1284ef3bdd6ac96739d0b7a: Generated new TSK 0
I20260806 10:50:24.610786 30371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:50:25.286895 30340 catalog_manager.cc:1259] Loaded cluster ID: 3afc7494d2ca40d984ec3a7da28e27ff
I20260806 10:50:25.286995 30340 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: loading cluster ID for follower catalog manager: success
I20260806 10:50:25.289546 30340 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: acquiring CA information for follower catalog manager: success
I20260806 10:50:25.291288 30340 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b0ab7d5062b0461ab3f724c1295dbc0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:50:25.338645 30354 catalog_manager.cc:1259] Loaded cluster ID: 3afc7494d2ca40d984ec3a7da28e27ff
I20260806 10:50:25.338769 30354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: loading cluster ID for follower catalog manager: success
I20260806 10:50:25.341251 30354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: acquiring CA information for follower catalog manager: success
I20260806 10:50:25.343310 30354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 727b1210673445809a08a54b4059b466: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:50:25.756740 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30098
I20260806 10:50:25.758347 30161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:25.871097 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30098
I20260806 10:50:25.896100 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30173
I20260806 10:50:25.897248 30235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:26.007261 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30173
W20260806 10:50:26.031567 30265 connection.cc:535] server connection from 127.0.0.1:51800 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 10:50:26.031872 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30247
I20260806 10:50:26.032950 30311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:26.154826 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30247
2026-08-06T10:50:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4860 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:50:26 dist-test-slave-0b58 krb5kdc[30390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:50:26 dist-test-slave-0b58 krb5kdc[30390](info): set up 2 sockets
Aug 06 10:50:26 dist-test-slave-0b58 krb5kdc[30390](info): commencing operation
krb5kdc: starting...
W20260806 10:50:27.296223 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:50:27 dist-test-slave-0b58 krb5kdc[30390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:50:27.563803 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45565
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:27.897140 30403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:27.897410 30403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:27.897534 30403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:27.904728 30403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:27.904824 30403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:27.904870 30403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:27.904924 30403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:27.904960 30403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:27.914637 30403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:27.915743 30403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:27.917018 30403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:27.928707 30408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:27.928732 30409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:27.928692 30411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:27.929876 30403 server_base.cc:1034] running on GCE node
I20260806 10:50:27.930541 30403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:27.931738 30403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:27.932933 30403 hybrid_clock.cc:648] HybridClock initialized: now 1786013427932898 us; error 25 us; skew 500 ppm
I20260806 10:50:27.933216 30403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:27 dist-test-slave-0b58 krb5kdc[30390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013427, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:27.936199 30403 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:27.937738 30403 webserver.cc:460] Webserver started at http://127.16.21.62:37613/ using document root <none> and password file <none>
I20260806 10:50:27.938385 30403 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:27.938539 30403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:27.938839 30403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:27.941731 30403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "92b02d556aef45cb833aa55c015fcb43"
format_stamp: "Formatted at 2026-08-06 10:50:27 on dist-test-slave-0b58"
server_key: "61b4573ff5c357ff5cad8cdcbb13b743"
server_key_iv: "66c26bde68675669ba5c7bb51d24e224"
server_key_version: "encryptionkey@0"
I20260806 10:50:27.942469 30403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "92b02d556aef45cb833aa55c015fcb43"
format_stamp: "Formatted at 2026-08-06 10:50:27 on dist-test-slave-0b58"
server_key: "61b4573ff5c357ff5cad8cdcbb13b743"
server_key_iv: "66c26bde68675669ba5c7bb51d24e224"
server_key_version: "encryptionkey@0"
I20260806 10:50:27.947408 30403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 10:50:27.950683 30418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:27.951864 30403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:50:27.952040 30403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "92b02d556aef45cb833aa55c015fcb43"
format_stamp: "Formatted at 2026-08-06 10:50:27 on dist-test-slave-0b58"
server_key: "61b4573ff5c357ff5cad8cdcbb13b743"
server_key_iv: "66c26bde68675669ba5c7bb51d24e224"
server_key_version: "encryptionkey@0"
I20260806 10:50:27.952232 30403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:27.966749 30403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:27.970446 30403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:27.970726 30403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:27.989461 30403 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:45565
I20260806 10:50:27.989504 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:45565 every 8 connection(s)
I20260806 10:50:27.990731 30403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:27.994263 30471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:27.997959 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30403
I20260806 10:50:27.998219 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:50:27.999111 16468 external_mini_cluster.cc:1428] Setting key 4b9e7d15dfe97dd57687a6f691399d69
I20260806 10:50:28.007504 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43: Bootstrap starting.
I20260806 10:50:28.011281 30471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:28.012605 30471 log.cc:824] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:50:28 dist-test-slave-0b58 krb5kdc[30390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:28.015355 30471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43: No bootstrap required, opened a new log
I20260806 10:50:28.022326 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:28.000984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.16.21.62:45565)
0806 10:50:28.001755 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:28.001766 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:28.001810 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:50:28.003427 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:28.003447 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:28.003831 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:28.004135 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:28.005979 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:28.007245 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:28.009182 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:28.009855 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:28.009977 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:28.015055 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:28.015063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:28.015075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:28.015107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:28.018357 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:28.019050 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:28.019056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:28.019061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:28.019228 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:28.019631 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:28.019635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:28.019637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:28.020157 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:28.020536 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:28.020998 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 10:50:28.021406 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":389,"threads_started":1}
I20260806 10:50:28.023500 30471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b02d556aef45cb833aa55c015fcb43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45565 } }
I20260806 10:50:28.023849 30471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:28.023976 30471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b02d556aef45cb833aa55c015fcb43, State: Initialized, Role: FOLLOWER
I20260806 10:50:28.024603 30471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b02d556aef45cb833aa55c015fcb43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45565 } }
I20260806 10:50:28.024891 30471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:28.025050 30471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:28.025224 30471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:28.026203 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:50:28.027598 30471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b02d556aef45cb833aa55c015fcb43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45565 } }
I20260806 10:50:28.028055 30471 leader_election.cc:302] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b02d556aef45cb833aa55c015fcb43; no voters:
I20260806 10:50:28.028786 30471 leader_election.cc:288] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:50:28.028888 30476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:28.030053 30476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [term 1 LEADER]: Becoming Leader. State: Replica: 92b02d556aef45cb833aa55c015fcb43, State: Running, Role: LEADER
I20260806 10:50:28.030649 30476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b02d556aef45cb833aa55c015fcb43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45565 } }
I20260806 10:50:28.031095 30471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:28.033571 30478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b02d556aef45cb833aa55c015fcb43. Latest consensus state: current_term: 1 leader_uuid: "92b02d556aef45cb833aa55c015fcb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b02d556aef45cb833aa55c015fcb43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45565 } } }
I20260806 10:50:28.033975 30478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:28.033731 30477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b02d556aef45cb833aa55c015fcb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b02d556aef45cb833aa55c015fcb43" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45565 } } }
I20260806 10:50:28.034118 30477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:28.037385 30483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:28.043195 30483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:28.051332 30483 catalog_manager.cc:1347] Generated new cluster ID: 27becad6204a49d58144d18ba431e0b9
I20260806 10:50:28.051435 30483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:28.068639 30483 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:50:28.069934 30483 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:28.082294 30483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92b02d556aef45cb833aa55c015fcb43: Generated new TSK 0
I20260806 10:50:28.083071 30483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:50:28.138667 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30403
I20260806 10:50:28.139796 30466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:28.239012 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30403
2026-08-06T10:50:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2080 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:50:28 dist-test-slave-0b58 krb5kdc[30499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:50:28 dist-test-slave-0b58 krb5kdc[30499](info): set up 2 sockets
Aug 06 10:50:28 dist-test-slave-0b58 krb5kdc[30499](info): commencing operation
krb5kdc: starting...
W20260806 10:50:29.364565 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:50:29 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:50:29.651613 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40035
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:29.981156 30512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:29.981479 30512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:29.981642 30512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:29.988883 30512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:29.989017 30512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:29.989130 30512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:29.989228 30512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:29.989318 30512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:29.999002 30512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:30.000283 30512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:30.001540 30512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:30.013175 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:30.013219 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:30.013262 30520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:30.014082 30512 server_base.cc:1034] running on GCE node
I20260806 10:50:30.014833 30512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:30.016167 30512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:30.017441 30512 hybrid_clock.cc:648] HybridClock initialized: now 1786013430017406 us; error 49 us; skew 500 ppm
I20260806 10:50:30.017882 30512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:30 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013430, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:30.020946 30512 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:30.022328 30512 webserver.cc:460] Webserver started at http://127.16.21.62:41065/ using document root <none> and password file <none>
I20260806 10:50:30.023007 30512 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:30.023151 30512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:30.023463 30512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:30.026443 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "5d4c44064a2e42a2a138bbea48f10fb7"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "920ed875fbdf75c1328f1e9d1413c004"
server_key_iv: "0125b164f2c7eb1b4a2b56e7846e695b"
server_key_version: "encryptionkey@0"
I20260806 10:50:30.027190 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "5d4c44064a2e42a2a138bbea48f10fb7"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "920ed875fbdf75c1328f1e9d1413c004"
server_key_iv: "0125b164f2c7eb1b4a2b56e7846e695b"
server_key_version: "encryptionkey@0"
I20260806 10:50:30.032081 30512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 10:50:30.035339 30527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:30.036670 30512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:30.036850 30512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "5d4c44064a2e42a2a138bbea48f10fb7"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "920ed875fbdf75c1328f1e9d1413c004"
server_key_iv: "0125b164f2c7eb1b4a2b56e7846e695b"
server_key_version: "encryptionkey@0"
I20260806 10:50:30.037041 30512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:30.062638 30512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:30.066429 30512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:30.066720 30512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:30.085402 30512 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40035
I20260806 10:50:30.085477 30579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40035 every 8 connection(s)
I20260806 10:50:30.086692 30512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:30.087234 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30512
I20260806 10:50:30.087489 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:50:30.088424 16468 external_mini_cluster.cc:1428] Setting key b824f25fd1f55feb18a534b73e39ea2e
I20260806 10:50:30.090960 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:30.101207 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap starting.
Aug 06 10:50:30 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:30.104887 30580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:30.106012 30580 log.cc:824] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:30.109172 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: No bootstrap required, opened a new log
I20260806 10:50:30.110546 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:30.090367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.16.21.62:40035)
0806 10:50:30.091189 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:30.091200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:30.091242 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:50:30.092987 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:30.093007 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:30.093352 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:30.093724 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:30.095187 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:30.096568 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:30.097847 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:30.098539 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:30.098739 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:30.103437 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:30.103448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:30.103459 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:30.103492 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:30.106692 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:30.107376 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:30.107383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:30.107388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:30.107543 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:30.107951 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:30.107954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:30.107956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:30.108503 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:30.108721 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:30.109197 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 10:50:30.109606 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":358,"threads_started":1}
I20260806 10:50:30.117275 30580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:30.117733 30580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:30.117830 30580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c44064a2e42a2a138bbea48f10fb7, State: Initialized, Role: FOLLOWER
I20260806 10:50:30.118363 30580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:30.118590 30580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:30.118741 30580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:30.118913 30580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:30.121034 30580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:30.121496 30580 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4c44064a2e42a2a138bbea48f10fb7; no voters:
I20260806 10:50:30.122079 30580 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:50:30.122211 30585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:30.122980 30585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4c44064a2e42a2a138bbea48f10fb7, State: Running, Role: LEADER
I20260806 10:50:30.123546 30585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:30.124148 30580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:30.128484 30586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } } }
I20260806 10:50:30.128594 30587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4c44064a2e42a2a138bbea48f10fb7. Latest consensus state: current_term: 1 leader_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } } }
I20260806 10:50:30.128937 30586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:30.129024 30587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:30.130947 30594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:30.135954 30594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:30.143819 30594 catalog_manager.cc:1347] Generated new cluster ID: 759e700790204cfd9c2af943ea3ec599
I20260806 10:50:30.143918 30594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:30.161684 30594 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:50:30.163475 30594 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:30.171640 30594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Generated new TSK 0
I20260806 10:50:30.172618 30594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:50:30.245249 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40035
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:30.576442 30606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:30.576699 30606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:30.576850 30606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:30.584013 30606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:30.584108 30606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:30.584259 30606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:50:30.593643 30606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:30.594847 30606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:30.596128 30606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:30.609684 30614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:30.609736 30611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:30.609717 30612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:30.610193 30606 server_base.cc:1034] running on GCE node
I20260806 10:50:30.610996 30606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:30.612002 30606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:30.613226 30606 hybrid_clock.cc:648] HybridClock initialized: now 1786013430613161 us; error 61 us; skew 500 ppm
I20260806 10:50:30.613566 30606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:30 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013430, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:30.616693 30606 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:50:30.618033 30606 webserver.cc:460] Webserver started at http://127.16.21.1:41065/ using document root <none> and password file <none>
I20260806 10:50:30.618693 30606 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:30.618842 30606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:30.619138 30606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:30.621965 30606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "ee4c4f051d624ad1bed97dce3098b190"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "d3c8d607afe9226f12a0755ab31b6012"
server_key_iv: "890110ce0b8db3f0636f1c3eb2b74fca"
server_key_version: "encryptionkey@0"
I20260806 10:50:30.622730 30606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4c4f051d624ad1bed97dce3098b190"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "d3c8d607afe9226f12a0755ab31b6012"
server_key_iv: "890110ce0b8db3f0636f1c3eb2b74fca"
server_key_version: "encryptionkey@0"
I20260806 10:50:30.627725 30606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 10:50:30.631336 30621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:30.632665 30606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:30.632838 30606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "ee4c4f051d624ad1bed97dce3098b190"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "d3c8d607afe9226f12a0755ab31b6012"
server_key_iv: "890110ce0b8db3f0636f1c3eb2b74fca"
server_key_version: "encryptionkey@0"
I20260806 10:50:30.633033 30606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:30.646669 30606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:30.650364 30606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:30.650677 30606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:30.651579 30606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:30.653003 30606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:30.653106 30606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:30.653249 30606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:30.653340 30606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:30.686585 30606 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:45515
I20260806 10:50:30.686638 30735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:45515 every 8 connection(s)
I20260806 10:50:30.687803 30606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:50:30.693328 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30606
I20260806 10:50:30.693558 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:50:30.694432 16468 external_mini_cluster.cc:1428] Setting key f9e2fc2d85c30845388a5f7099314a38
Aug 06 10:50:30 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013430, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:30.710026 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:30.690306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:52751 (local address 127.16.21.62:40035)
0806 10:50:30.690784 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:30.690791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:30.691686 (+ 895us) server_negotiation.cc:407] Connection header received
0806 10:50:30.693859 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:30.693864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:30.693938 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:30.694031 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:30.695771 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:30.696763 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:30.699196 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:30.699558 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:30.699649 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:30.704097 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:30.704102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:30.704105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:30.704137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:30.706776 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:30.707687 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:30.707691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:30.707693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:30.707814 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:30.708356 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:30.708359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:30.708361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:30.708617 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:30.708725 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:30.709596 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 10:50:30.709760 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260806 10:50:30.711079 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:30.690800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:52751)
0806 10:50:30.691532 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:30.691574 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:30.693568 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:30.694325 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:30.694335 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:30.695057 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:30.695610 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:30.695626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:30.696928 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:30.696933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:30.699043 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:30.699062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:30.699780 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:30.699790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:30.700105 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:30.700863 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:30.700896 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:30.703876 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:30.706922 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:30.706933 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:30.706951 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:30.707569 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:30.707923 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:30.707930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:30.707936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:30.708190 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:30.708742 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:30.708752 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:30.709182 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 10:50:30.709482 (+ 300us) client_negotiation.cc:240] Negotiation successful
0806 10:50:30.710031 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":286,"threads_started":1}
I20260806 10:50:30.712687 30736 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40035
I20260806 10:50:30.712987 30736 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:30.713686 30736 heartbeater.cc:511] Master 127.16.21.62:40035 requested a full tablet report, sending...
I20260806 10:50:30.715663 30544 ts_manager.cc:194] Registered new tserver with Master: ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515)
I20260806 10:50:30.717193 30544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:52751
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:50:30.759764 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40035
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:31.105114 30742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:31.105417 30742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:31.105592 30742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:31.112936 30742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:31.113073 30742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:31.113270 30742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:50:31.123075 30742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:31.124311 30742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:31.125595 30742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:31.138271 30747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:31.138237 30748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:31.138298 30750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:31.139657 30742 server_base.cc:1034] running on GCE node
I20260806 10:50:31.140163 30742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:31.141137 30742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:31.142343 30742 hybrid_clock.cc:648] HybridClock initialized: now 1786013431142294 us; error 34 us; skew 500 ppm
I20260806 10:50:31.142758 30742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:31 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013431, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:31.145905 30742 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:50:31.147285 30742 webserver.cc:460] Webserver started at http://127.16.21.2:39381/ using document root <none> and password file <none>
I20260806 10:50:31.147943 30742 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:31.148085 30742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:31.148455 30742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:31.151182 30742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "41a67f43330546adac2b17c4c9d78af0"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "8875c3b2745c7030b935910a7bfa4714"
server_key_iv: "cc45b7b9313e35a5b0d140e053cc5f7c"
server_key_version: "encryptionkey@0"
I20260806 10:50:31.151950 30742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "41a67f43330546adac2b17c4c9d78af0"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "8875c3b2745c7030b935910a7bfa4714"
server_key_iv: "cc45b7b9313e35a5b0d140e053cc5f7c"
server_key_version: "encryptionkey@0"
I20260806 10:50:31.156787 30742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 10:50:31.160043 30757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:31.161363 30742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:31.161553 30742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "41a67f43330546adac2b17c4c9d78af0"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "8875c3b2745c7030b935910a7bfa4714"
server_key_iv: "cc45b7b9313e35a5b0d140e053cc5f7c"
server_key_version: "encryptionkey@0"
I20260806 10:50:31.161785 30742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:31.175127 30742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:31.178838 30742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:31.179121 30742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:31.179996 30742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:31.181592 30742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:31.181713 30742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:31.181872 30742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:31.181962 30742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:31.215499 30742 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:43471
I20260806 10:50:31.215581 30870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:43471 every 8 connection(s)
I20260806 10:50:31.216815 30742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:50:31.224717 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30742
I20260806 10:50:31.224925 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:50:31.225637 16468 external_mini_cluster.cc:1428] Setting key a25fe9985e765a1a931fbb2051d06d3e
Aug 06 10:50:31 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013431, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:31.242496 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.219327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:40313 (local address 127.16.21.62:40035)
0806 10:50:31.219779 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:31.219785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:31.220760 (+ 975us) server_negotiation.cc:407] Connection header received
0806 10:50:31.222589 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:31.222595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:31.222706 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:31.222829 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:31.225415 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.226643 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.229122 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.229845 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.229995 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.234406 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:31.234413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:31.234416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:31.234450 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:31.238367 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:31.239347 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:31.239356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:31.239362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:31.239527 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:31.240162 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:31.240169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:31.240175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:31.240598 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:31.240748 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:31.241746 (+ 998us) server_negotiation.cc:299] Negotiation successful
0806 10:50:31.242000 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20260806 10:50:31.243983 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.219871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:40313)
0806 10:50:31.220608 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:31.220650 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:31.222237 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:31.223439 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:31.223449 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:31.224334 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:31.225223 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.225246 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.226849 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.226856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.228942 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.228959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.230126 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.230138 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.230508 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.231262 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:31.231297 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:31.234126 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:31.238523 (+ 4397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:31.238532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:31.238552 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:31.239202 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:31.239651 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:31.239658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:31.239665 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:31.240011 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:31.240823 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:31.240833 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:31.241294 (+ 461us) client_negotiation.cc:769] Sending connection context
0806 10:50:31.241593 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 10:50:31.242155 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":318,"threads_started":1}
I20260806 10:50:31.245613 30871 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40035
I20260806 10:50:31.245918 30871 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:31.246647 30871 heartbeater.cc:511] Master 127.16.21.62:40035 requested a full tablet report, sending...
I20260806 10:50:31.248135 30544 ts_manager.cc:194] Registered new tserver with Master: 41a67f43330546adac2b17c4c9d78af0 (127.16.21.2:43471)
I20260806 10:50:31.249446 30544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:40313
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:50:31.293356 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40035
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:31.617702 30877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:31.617950 30877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:31.618101 30877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:31.625106 30877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:31.625242 30877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:31.625379 30877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:50:31.634505 30877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:31.635644 30877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:31.636938 30877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:31.649978 30885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:31.649930 30883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:31.650017 30882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:31.651042 30877 server_base.cc:1034] running on GCE node
I20260806 10:50:31.651633 30877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:31.652573 30877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:31.653766 30877 hybrid_clock.cc:648] HybridClock initialized: now 1786013431653730 us; error 24 us; skew 500 ppm
I20260806 10:50:31.654069 30877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:31 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013431, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:31.657212 30877 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:50:31.658540 30877 webserver.cc:460] Webserver started at http://127.16.21.3:37599/ using document root <none> and password file <none>
I20260806 10:50:31.659124 30877 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:31.659229 30877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:31.659559 30877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:31.662299 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "2380a6ddb6754395b7f51b3baa326213"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "87d8c3ed74e8503d4e31c12248ab58ff"
server_key_iv: "425a7cb9b0cf41677ba432ca86cc7fe5"
server_key_version: "encryptionkey@0"
I20260806 10:50:31.662982 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "2380a6ddb6754395b7f51b3baa326213"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "87d8c3ed74e8503d4e31c12248ab58ff"
server_key_iv: "425a7cb9b0cf41677ba432ca86cc7fe5"
server_key_version: "encryptionkey@0"
I20260806 10:50:31.667904 30877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:50:31.671059 30893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:31.672344 30877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:31.672510 30877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "2380a6ddb6754395b7f51b3baa326213"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "87d8c3ed74e8503d4e31c12248ab58ff"
server_key_iv: "425a7cb9b0cf41677ba432ca86cc7fe5"
server_key_version: "encryptionkey@0"
I20260806 10:50:31.672700 30877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:31.692433 30877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:31.696411 30877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:31.696681 30877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:31.697546 30877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:31.698750 30877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:31.698854 30877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:31.699005 30877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:31.699081 30877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:31.719861 30736 heartbeater.cc:503] Master 127.16.21.62:40035 was elected leader, sending a full tablet report...
I20260806 10:50:31.732501 30877 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:33993
I20260806 10:50:31.732574 31005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:33993 every 8 connection(s)
I20260806 10:50:31.733829 30877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:50:31.735936 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 30877
I20260806 10:50:31.736166 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:50:31.736774 16468 external_mini_cluster.cc:1428] Setting key adf2e9c75ec27a17641beb08628172d5
Aug 06 10:50:31 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013431, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:31.753466 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.736377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:45657 (local address 127.16.21.62:40035)
0806 10:50:31.736604 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:31.736611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:31.737953 (+ 1342us) server_negotiation.cc:407] Connection header received
0806 10:50:31.740303 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:31.740309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:31.740395 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:31.740477 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:31.741944 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.742723 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.744163 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.744606 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.744701 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.748008 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:31.748013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:31.748016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:31.748046 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:31.750495 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:31.751353 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:31.751358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:31.751361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:31.751493 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:31.752010 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:31.752016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:31.752019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:31.752362 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:31.752466 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:31.753069 (+ 603us) server_negotiation.cc:299] Negotiation successful
0806 10:50:31.753243 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 10:50:31.754335 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.736924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:45657)
0806 10:50:31.737786 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:31.737836 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:31.740007 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:31.740736 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:31.740743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:31.741229 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:31.741779 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.741806 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.742858 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.742862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.744048 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.744059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.744787 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.744792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.744972 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.745642 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:31.745668 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:31.747816 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:31.750620 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:31.750630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:31.750645 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:31.751241 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:31.751620 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:31.751624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:31.751627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:31.751889 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:31.752480 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:31.752485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:31.752761 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 10:50:31.752973 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 10:50:31.753309 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":313,"threads_started":1}
I20260806 10:50:31.755699 31006 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40035
I20260806 10:50:31.755977 31006 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:31.756737 31006 heartbeater.cc:511] Master 127.16.21.62:40035 requested a full tablet report, sending...
I20260806 10:50:31.758068 30544 ts_manager.cc:194] Registered new tserver with Master: 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993)
I20260806 10:50:31.758857 30544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:45657
I20260806 10:50:31.763988 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:50:31.781164 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.768529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.16.21.62:40035)
0806 10:50:31.768726 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:31.768733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:31.769071 (+ 338us) server_negotiation.cc:407] Connection header received
0806 10:50:31.769259 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:31.769264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:31.769343 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:31.769445 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:31.770810 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.771522 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.772901 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.773318 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.773445 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.775840 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:31.775845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:31.775849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:31.775877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:31.778558 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:31.779216 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:31.779223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:31.779227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:31.779344 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:31.779730 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:31.779735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:31.779739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:31.780035 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:31.780343 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:31.780623 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 10:50:31.780840 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:50:31.784911 30544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:50:31.791620 30544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:50:31.838554 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.802853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33993 (local address 127.0.0.1:47440)
0806 10:50:31.828203 (+ 25350us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:31.828275 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:31.828465 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:31.831331 (+ 2866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:31.831336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:31.831361 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:31.831794 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.831807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.833341 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.833348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.835702 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.835713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.837534 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.837541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.837729 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.837793 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 10:50:31.837936 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 10:50:31.838126 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25051,"mutex_wait_us":127,"thread_start_us":25905,"threads_started":1}
I20260806 10:50:31.838724 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.804578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.16.21.1:45515)
0806 10:50:31.826660 (+ 22082us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:31.826673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:31.829856 (+ 3183us) server_negotiation.cc:407] Connection header received
0806 10:50:31.830029 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:31.830038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:31.830290 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:31.830501 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:31.831363 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.833114 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.836175 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.837626 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.837754 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.837967 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:31.838250 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 10:50:31.838420 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21853,"thread_start_us":22100,"threads_started":1}
I20260806 10:50:31.839143 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.803852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.16.21.2:43471)
0806 10:50:31.827372 (+ 23520us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:31.827383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:31.829249 (+ 1866us) server_negotiation.cc:407] Connection header received
0806 10:50:31.829448 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:31.829458 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:31.829797 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:31.830023 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:31.831329 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.833257 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.836310 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.837533 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.837658 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.837831 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:31.838656 (+ 825us) server_negotiation.cc:299] Negotiation successful
0806 10:50:31.838809 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23319,"thread_start_us":23517,"threads_started":1}
I20260806 10:50:31.839623 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.803373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:43471 (local address 127.0.0.1:34586)
0806 10:50:31.829061 (+ 25688us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:31.829083 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:31.829226 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:31.830414 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:31.830422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:31.830455 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:31.831100 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.831118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.833378 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.833388 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.836087 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.836145 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.837986 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.837992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.838156 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.838184 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:50:31.838286 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 10:50:31.838402 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25543,"mutex_wait_us":60,"spinlock_wait_cycles":67328,"thread_start_us":26102,"threads_started":1}
I20260806 10:50:31.839659 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.803805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:45515 (local address 127.0.0.1:48788)
0806 10:50:31.829716 (+ 25911us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:31.829728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:31.829887 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:31.830580 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:31.830585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:31.830634 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:31.831100 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.831118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.833255 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.833261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.836001 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:31.836017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.837865 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.837872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:31.838061 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.838099 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:50:31.838200 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:50:31.838320 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25812,"mutex_wait_us":95,"thread_start_us":23940,"threads_started":1}
I20260806 10:50:31.840052 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:31.803149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.16.21.3:33993)
0806 10:50:31.830463 (+ 27314us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:31.830471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:31.830500 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:50:31.830591 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:31.830598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:31.830805 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:31.831018 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:31.831996 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.833142 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.835847 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:31.837330 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:31.837461 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:31.837697 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:31.839172 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0806 10:50:31.839670 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27135,"thread_start_us":26951,"threads_started":1}
I20260806 10:50:31.843134 30805 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8d47fc7c7946fda3119dc7c0bca20d (DEFAULT_TABLE table=test-table [id=8bade5ba5564460daf0449869629f96a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:31.843423 30941 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8d47fc7c7946fda3119dc7c0bca20d (DEFAULT_TABLE table=test-table [id=8bade5ba5564460daf0449869629f96a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:31.843439 30670 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8d47fc7c7946fda3119dc7c0bca20d (DEFAULT_TABLE table=test-table [id=8bade5ba5564460daf0449869629f96a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:31.844566 30805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8d47fc7c7946fda3119dc7c0bca20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:31.844836 30670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8d47fc7c7946fda3119dc7c0bca20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:31.844861 30941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8d47fc7c7946fda3119dc7c0bca20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:31.855336 31025 tablet_bootstrap.cc:492] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Bootstrap starting.
I20260806 10:50:31.858336 31025 tablet_bootstrap.cc:654] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:31.859560 31025 log.cc:824] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:31.859619 31026 tablet_bootstrap.cc:492] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Bootstrap starting.
I20260806 10:50:31.861444 31027 tablet_bootstrap.cc:492] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Bootstrap starting.
I20260806 10:50:31.863893 31026 tablet_bootstrap.cc:654] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:31.864770 31027 tablet_bootstrap.cc:654] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:31.865211 31026 log.cc:824] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:31.865675 31025 tablet_bootstrap.cc:492] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: No bootstrap required, opened a new log
I20260806 10:50:31.865981 31025 ts_tablet_manager.cc:1397] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260806 10:50:31.866135 31027 log.cc:824] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:31.868546 31026 tablet_bootstrap.cc:492] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: No bootstrap required, opened a new log
I20260806 10:50:31.868793 31026 ts_tablet_manager.cc:1397] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260806 10:50:31.874192 31027 tablet_bootstrap.cc:492] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: No bootstrap required, opened a new log
I20260806 10:50:31.874444 31027 ts_tablet_manager.cc:1397] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260806 10:50:31.876449 31025 raft_consensus.cc:348] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:31.876889 31025 raft_consensus.cc:374] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:31.876989 31025 raft_consensus.cc:729] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a67f43330546adac2b17c4c9d78af0, State: Initialized, Role: FOLLOWER
I20260806 10:50:31.877619 31025 consensus_queue.cc:260] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:31.878757 30871 heartbeater.cc:503] Master 127.16.21.62:40035 was elected leader, sending a full tablet report...
I20260806 10:50:31.879366 31025 ts_tablet_manager.cc:1428] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260806 10:50:31.880162 31026 raft_consensus.cc:348] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:31.880843 31026 raft_consensus.cc:374] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:31.880951 31026 raft_consensus.cc:729] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4c4f051d624ad1bed97dce3098b190, State: Initialized, Role: FOLLOWER
I20260806 10:50:31.881522 31026 consensus_queue.cc:260] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:31.883302 31026 ts_tablet_manager.cc:1428] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260806 10:50:31.884661 31027 raft_consensus.cc:348] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:31.885022 31027 raft_consensus.cc:374] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:31.885174 31027 raft_consensus.cc:729] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2380a6ddb6754395b7f51b3baa326213, State: Initialized, Role: FOLLOWER
I20260806 10:50:31.885730 31027 consensus_queue.cc:260] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:31.886839 31006 heartbeater.cc:503] Master 127.16.21.62:40035 was elected leader, sending a full tablet report...
I20260806 10:50:31.887388 31027 ts_tablet_manager.cc:1428] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
W20260806 10:50:31.940253 30737 tablet.cc:2369] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:50:31.968909 30872 tablet.cc:2369] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:50:31.985379 31007 tablet.cc:2369] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:32.001240 31031 raft_consensus.cc:482] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:50:32.001506 31031 raft_consensus.cc:504] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:32.003075 31031 leader_election.cc:288] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993), ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515)
I20260806 10:50:32.010215 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.003621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:45075 (local address 127.16.21.1:45515)
0806 10:50:32.003856 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:32.003865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:32.004561 (+ 696us) server_negotiation.cc:407] Connection header received
0806 10:50:32.004722 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:32.004727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:32.004819 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:32.004902 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:32.005548 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.006387 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.008287 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.009124 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.009230 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.009357 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:32.009794 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 10:50:32.009927 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:50:32.010247 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.003549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:57591 (local address 127.16.21.3:33993)
0806 10:50:32.003758 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:32.003765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:32.004700 (+ 935us) server_negotiation.cc:407] Connection header received
0806 10:50:32.004857 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:32.004860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:32.004950 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:32.005044 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:32.005734 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.006631 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.008416 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.009180 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.009338 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.009440 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:32.009851 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 10:50:32.010002 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:50:32.010402 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.003580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:45515 (local address 127.16.21.2:45075)
0806 10:50:32.004422 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:32.004446 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:32.004581 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:32.004942 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:32.004947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:32.004971 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:32.005387 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:32.005396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.006515 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.006520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:32.008085 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:32.008097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.009385 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.009391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:32.009621 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.009646 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 10:50:32.009722 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:50:32.009852 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":17,"spinlock_wait_cycles":23936,"thread_start_us":382,"threads_started":1}
I20260806 10:50:32.010586 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.003664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33993 (local address 127.16.21.2:57591)
0806 10:50:32.004549 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:32.004568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:32.004733 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:32.005083 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:32.005087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:32.005136 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:32.005560 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:32.005571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.006775 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.006780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:32.008225 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:32.008242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.009440 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.009444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:32.009668 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.009691 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:50:32.009809 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 10:50:32.009927 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":57,"thread_start_us":465,"threads_started":1}
I20260806 10:50:32.011518 30690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8d47fc7c7946fda3119dc7c0bca20d" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4c4f051d624ad1bed97dce3098b190" is_pre_election: true
I20260806 10:50:32.011631 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8d47fc7c7946fda3119dc7c0bca20d" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2380a6ddb6754395b7f51b3baa326213" is_pre_election: true
I20260806 10:50:32.012076 30690 raft_consensus.cc:2454] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 0.
I20260806 10:50:32.012118 30961 raft_consensus.cc:2454] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 0.
I20260806 10:50:32.012869 30758 leader_election.cc:302] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2380a6ddb6754395b7f51b3baa326213, 41a67f43330546adac2b17c4c9d78af0; no voters:
I20260806 10:50:32.013289 31031 raft_consensus.cc:2781] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:50:32.013408 31031 raft_consensus.cc:482] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:50:32.013479 31031 raft_consensus.cc:3037] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:32.015589 31031 raft_consensus.cc:504] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:32.016139 31031 leader_election.cc:288] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 election: Requested vote from peers 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993), ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515)
I20260806 10:50:32.016605 30690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8d47fc7c7946fda3119dc7c0bca20d" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4c4f051d624ad1bed97dce3098b190"
I20260806 10:50:32.016601 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8d47fc7c7946fda3119dc7c0bca20d" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2380a6ddb6754395b7f51b3baa326213"
I20260806 10:50:32.016820 30690 raft_consensus.cc:3037] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:32.016814 30961 raft_consensus.cc:3037] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:32.020036 30961 raft_consensus.cc:2454] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 1.
I20260806 10:50:32.020037 30690 raft_consensus.cc:2454] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 1.
I20260806 10:50:32.020447 30761 leader_election.cc:302] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a67f43330546adac2b17c4c9d78af0, ee4c4f051d624ad1bed97dce3098b190; no voters:
I20260806 10:50:32.020797 31031 raft_consensus.cc:2781] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:32.021239 31031 raft_consensus.cc:686] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 1 LEADER]: Becoming Leader. State: Replica: 41a67f43330546adac2b17c4c9d78af0, State: Running, Role: LEADER
I20260806 10:50:32.021812 31031 consensus_queue.cc:237] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } }
I20260806 10:50:32.026623 30544 catalog_manager.cc:5495] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41a67f43330546adac2b17c4c9d78af0 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "41a67f43330546adac2b17c4c9d78af0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20260806 10:50:32.080370 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.074938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.16.21.2:43471)
0806 10:50:32.075170 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:32.075177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:32.075196 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:50:32.075310 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:32.075314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:32.075397 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:32.075544 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:50:32.075906 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.076699 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.078136 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.078546 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.078682 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.079257 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:50:32.079722 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:50:32.079889 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:32.079945 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 10:50:32.080022 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 10:50:32.088304 30690 raft_consensus.cc:1261] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Refusing update from remote peer 41a67f43330546adac2b17c4c9d78af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:32.088392 30961 raft_consensus.cc:1261] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Refusing update from remote peer 41a67f43330546adac2b17c4c9d78af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:32.089321 31036 consensus_queue.cc:1035] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:32.089694 31031 consensus_queue.cc:1035] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:32.100934 31039 mvcc.cc:205] Tried to move back new op lower bound from 7315511017823768576 to 7315511017565638656. Current Snapshot: MvccSnapshot[applied={T|T < 7315511017823768576}]
I20260806 10:50:32.119433 30544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:32.119714 30544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:32.123891 30544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Sending DeleteTablet for 3 replicas of tablet 4f8d47fc7c7946fda3119dc7c0bca20d
I20260806 10:50:32.125005 30941 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8d47fc7c7946fda3119dc7c0bca20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:32 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:47440
I20260806 10:50:32.125375 30805 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8d47fc7c7946fda3119dc7c0bca20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:32 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:34586
I20260806 10:50:32.126060 31049 tablet_replica.cc:330] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: stopping tablet replica
I20260806 10:50:32.126555 31049 raft_consensus.cc:2229] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:32.126509 30670 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8d47fc7c7946fda3119dc7c0bca20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:32 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:48788
I20260806 10:50:32.127091 31049 raft_consensus.cc:2258] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:32.127514 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30606
I20260806 10:50:32.137912 31049 ts_tablet_manager.cc:1905] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:32.138379 31050 tablet_replica.cc:330] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: stopping tablet replica
I20260806 10:50:32.138795 31050 raft_consensus.cc:2229] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:50:32.139463 31050 raft_consensus.cc:2258] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:32.141058 31051 tablet_replica.cc:330] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: stopping tablet replica
I20260806 10:50:32.141677 31051 raft_consensus.cc:2229] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:32.142177 31051 raft_consensus.cc:2258] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:32.144591 31051 ts_tablet_manager.cc:1905] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:32.144609 31050 ts_tablet_manager.cc:1905] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:32.151674 31051 ts_tablet_manager.cc:1918] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:32.151839 31051 log.cc:1196] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/4f8d47fc7c7946fda3119dc7c0bca20d
I20260806 10:50:32.152240 31050 ts_tablet_manager.cc:1918] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:32.152452 31051 ts_tablet_manager.cc:1939] T 4f8d47fc7c7946fda3119dc7c0bca20d P ee4c4f051d624ad1bed97dce3098b190: Deleting consensus metadata
I20260806 10:50:32.152462 31050 log.cc:1196] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/4f8d47fc7c7946fda3119dc7c0bca20d
I20260806 10:50:32.153069 31050 ts_tablet_manager.cc:1939] T 4f8d47fc7c7946fda3119dc7c0bca20d P 41a67f43330546adac2b17c4c9d78af0: Deleting consensus metadata
I20260806 10:50:32.154345 30531 catalog_manager.cc:4841] TS ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515): tablet 4f8d47fc7c7946fda3119dc7c0bca20d (table test-table [id=8bade5ba5564460daf0449869629f96a]) successfully deleted
Aug 06 10:50:32 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:50:32.154778 30530 catalog_manager.cc:4841] TS 41a67f43330546adac2b17c4c9d78af0 (127.16.21.2:43471): tablet 4f8d47fc7c7946fda3119dc7c0bca20d (table test-table [id=8bade5ba5564460daf0449869629f96a]) successfully deleted
I20260806 10:50:32.158181 31049 ts_tablet_manager.cc:1918] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:32.158380 31049 log.cc:1196] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/4f8d47fc7c7946fda3119dc7c0bca20d
I20260806 10:50:32.158960 31049 ts_tablet_manager.cc:1939] T 4f8d47fc7c7946fda3119dc7c0bca20d P 2380a6ddb6754395b7f51b3baa326213: Deleting consensus metadata
I20260806 10:50:32.160651 30528 catalog_manager.cc:4841] TS 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993): tablet 4f8d47fc7c7946fda3119dc7c0bca20d (table test-table [id=8bade5ba5564460daf0449869629f96a]) successfully deleted
I20260806 10:50:32.162106 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.128787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.16.21.1:45515)
0806 10:50:32.129014 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:32.129023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:32.140826 (+ 11803us) server_negotiation.cc:407] Connection header received
0806 10:50:32.142891 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:32.142899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:32.142999 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:32.143109 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:32.145424 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.146624 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.148719 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.149410 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.149517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.155779 (+ 6262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:32.155790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:32.155804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:32.155834 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:32.159062 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:32.159766 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:32.159773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:32.159775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:32.159901 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:32.160362 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:32.160365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:32.160367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:32.160850 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:32.161184 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:32.161516 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 10:50:32.161755 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:50:32.162742 30731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:32.315192 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30606
I20260806 10:50:32.336938 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30742
Aug 06 10:50:32 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:50:32.353629 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.337735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.16.21.2:43471)
0806 10:50:32.337915 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:32.337923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:32.337969 (+ 46us) server_negotiation.cc:407] Connection header received
0806 10:50:32.338185 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:32.338196 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:32.338271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:32.338389 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:32.339805 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.340817 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.342231 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.342717 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.342847 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.347414 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:32.347428 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:32.347446 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:32.347480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:32.350652 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:32.351409 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:32.351425 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:32.351429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:32.351568 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:32.351966 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:32.351972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:32.351975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:32.352452 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:32.352816 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:32.353031 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 10:50:32.353256 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:50:32.354338 30866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:32.472400 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30742
I20260806 10:50:32.491731 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30877
Aug 06 10:50:32 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:50:32.506596 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:32.492592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.16.21.3:33993)
0806 10:50:32.492784 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:32.492789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:32.492807 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:50:32.492948 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:32.492952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:32.493026 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:32.493111 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:32.494389 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.495120 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.496510 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:32.496902 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:32.496999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:32.501441 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:32.501455 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:32.501470 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:32.501503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:32.504032 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:32.504763 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:32.504771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:32.504773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:32.504924 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:32.505272 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:32.505275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:32.505277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:32.505604 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:32.505820 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:32.506068 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 10:50:32.506236 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:50:32.507251 31001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:32.598663 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30877
I20260806 10:50:32.619537 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 30512
I20260806 10:50:32.620780 30575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:32.734261 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 30512
I20260806 10:50:32.759908 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40035
--webserver_interface=127.16.21.62
--webserver_port=41065
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:33.096489 31057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:33.096812 31057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:33.096979 31057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:33.104377 31057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:33.104522 31057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:33.104624 31057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:33.104729 31057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:33.104827 31057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:33.114682 31057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=41065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:33.116044 31057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:33.117373 31057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:33.144006 31066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:33.144066 31063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:33.144004 31064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:33.145195 31057 server_base.cc:1034] running on GCE node
I20260806 10:50:33.145962 31057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:33.147500 31057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:33.148746 31057 hybrid_clock.cc:648] HybridClock initialized: now 1786013433148692 us; error 53 us; skew 500 ppm
I20260806 10:50:33.149143 31057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:33 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:33.152678 31057 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:33.154289 31057 webserver.cc:460] Webserver started at http://127.16.21.62:41065/ using document root <none> and password file <none>
I20260806 10:50:33.155021 31057 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:33.155193 31057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:33.161192 31057 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:50:33.164021 31073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:33.166147 31057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:50:33.166412 31057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "5d4c44064a2e42a2a138bbea48f10fb7"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "920ed875fbdf75c1328f1e9d1413c004"
server_key_iv: "0125b164f2c7eb1b4a2b56e7846e695b"
server_key_version: "encryptionkey@0"
I20260806 10:50:33.167510 31057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:33.189780 31057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:33.194088 31057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:33.194419 31057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:33.214186 31057 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40035
I20260806 10:50:33.214208 31124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40035 every 8 connection(s)
I20260806 10:50:33.215711 31057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:33.219540 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31057
I20260806 10:50:33.220461 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:45515
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:40035
--webserver_port=41065
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:33.223125 31125 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:50:33.226464 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap starting.
I20260806 10:50:33.250510 31125 log.cc:824] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:33.262817 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:50:33.263363 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap complete.
I20260806 10:50:33.273350 31125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:33.274233 31125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c44064a2e42a2a138bbea48f10fb7, State: Initialized, Role: FOLLOWER
I20260806 10:50:33.274876 31125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:33.275146 31125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:33.275291 31125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:33.275480 31125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:50:33.279331 31125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:33.279892 31125 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4c44064a2e42a2a138bbea48f10fb7; no voters:
I20260806 10:50:33.280586 31125 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:50:33.280762 31129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:50:33.282035 31129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 LEADER]: Becoming Leader. State: Replica: 5d4c44064a2e42a2a138bbea48f10fb7, State: Running, Role: LEADER
I20260806 10:50:33.282589 31129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:33.282773 31125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:33.285406 31130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } } }
I20260806 10:50:33.285842 31130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:33.290594 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4c44064a2e42a2a138bbea48f10fb7. Latest consensus state: current_term: 2 leader_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } } }
I20260806 10:50:33.290899 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:33.294692 31134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:33.303185 31134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:33.304605 31134 catalog_manager.cc:1259] Loaded cluster ID: 759e700790204cfd9c2af943ea3ec599
I20260806 10:50:33.304749 31134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:33.307935 31134 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 10:50:33.326416 31134 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 10:50:33.335342 31134 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:33.337124 31134 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Loaded TSK: 0
I20260806 10:50:33.338827 31134 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:50:33.586457 31127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:33.586728 31127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:33.586889 31127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:33.594200 31127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:33.594307 31127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:33.594487 31127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:50:33.604418 31127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:45515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=41065
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:33.605717 31127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:33.607055 31127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:33.622037 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:33.622128 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:33.621959 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:33.623083 31127 server_base.cc:1034] running on GCE node
I20260806 10:50:33.623745 31127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:33.624861 31127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:33.626132 31127 hybrid_clock.cc:648] HybridClock initialized: now 1786013433626081 us; error 52 us; skew 500 ppm
I20260806 10:50:33.626500 31127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:33 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:33.634369 31127 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:50:33.636124 31127 webserver.cc:460] Webserver started at http://127.16.21.1:41065/ using document root <none> and password file <none>
I20260806 10:50:33.637013 31127 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:33.637168 31127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:33.644140 31127 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 10:50:33.647662 31163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:33.649277 31127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:33.649447 31127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "ee4c4f051d624ad1bed97dce3098b190"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "d3c8d607afe9226f12a0755ab31b6012"
server_key_iv: "890110ce0b8db3f0636f1c3eb2b74fca"
server_key_version: "encryptionkey@0"
I20260806 10:50:33.650831 31127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:33.685950 31127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:33.690001 31127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:33.690231 31127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:33.691176 31127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:33.692473 31127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:33.692571 31127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:33.692651 31127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:33.692708 31127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:33.726028 31127 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:45515
I20260806 10:50:33.726127 31276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:45515 every 8 connection(s)
I20260806 10:50:33.727510 31127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:50:33.738049 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31127
I20260806 10:50:33.738715 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:43471
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:40035
--webserver_port=39381
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:50:33 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013433, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:33.751614 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:33.730515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:35777)
0806 10:50:33.731310 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:33.731347 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:33.733075 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:33.734285 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:33.734293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:33.734772 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:33.735304 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:33.735320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:33.736981 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:33.736985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:33.738371 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:33.738388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:33.739493 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:33.739500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:33.739766 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:33.740497 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:33.740518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:33.742868 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:33.747617 (+ 4749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:33.747628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:33.747646 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:33.748095 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:33.748616 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:33.748619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:33.748621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:33.748819 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:33.749744 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:33.749749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:33.750035 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 10:50:33.750252 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 10:50:33.750642 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":340,"threads_started":1}
I20260806 10:50:33.752184 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:33.730413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:35777 (local address 127.16.21.62:40035)
0806 10:50:33.731103 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:33.731119 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:33.731476 (+ 357us) server_negotiation.cc:407] Connection header received
0806 10:50:33.733561 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:33.733573 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:33.733872 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:33.734158 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:33.735501 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:33.736831 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:33.738529 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:33.739244 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:33.739449 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:33.743152 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:33.743164 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:33.743187 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:33.743236 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:33.747441 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:33.748226 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:33.748238 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:33.748249 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:33.748495 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:33.748932 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:33.748940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:33.748946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:33.749608 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:33.749814 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:33.750500 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 10:50:33.750940 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":268,"threads_started":1}
I20260806 10:50:33.755290 31277 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40035
I20260806 10:50:33.755626 31277 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:33.756486 31277 heartbeater.cc:511] Master 127.16.21.62:40035 requested a full tablet report, sending...
I20260806 10:50:33.758625 31090 ts_manager.cc:194] Registered new tserver with Master: ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515)
I20260806 10:50:33.760071 31090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:35777
W20260806 10:50:34.079258 31281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:34.079504 31281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:34.079665 31281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:34.087028 31281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:34.087122 31281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:34.087340 31281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:50:34.096958 31281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:43471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=39381
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:34.098165 31281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:34.099438 31281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:34.112453 31286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:34.112360 31289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:34.112416 31287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:34.113365 31281 server_base.cc:1034] running on GCE node
I20260806 10:50:34.113942 31281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:34.114845 31281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:34.116084 31281 hybrid_clock.cc:648] HybridClock initialized: now 1786013434116038 us; error 46 us; skew 500 ppm
I20260806 10:50:34.116473 31281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:34 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013434, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:34.119499 31281 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:50:34.121011 31281 webserver.cc:460] Webserver started at http://127.16.21.2:39381/ using document root <none> and password file <none>
I20260806 10:50:34.121613 31281 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:34.121726 31281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:34.127337 31281 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:50:34.130213 31296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:34.131513 31281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:50:34.131681 31281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "41a67f43330546adac2b17c4c9d78af0"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "8875c3b2745c7030b935910a7bfa4714"
server_key_iv: "cc45b7b9313e35a5b0d140e053cc5f7c"
server_key_version: "encryptionkey@0"
I20260806 10:50:34.132716 31281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:34.155315 31281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:34.159404 31281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:34.159686 31281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:34.160768 31281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:34.162178 31281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:34.162267 31281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:34.162433 31281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:34.162535 31281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:34.197186 31281 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:43471
I20260806 10:50:34.197249 31409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:43471 every 8 connection(s)
I20260806 10:50:34.198679 31281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:50:34.202219 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31281
I20260806 10:50:34.202896 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:33993
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:40035
--webserver_port=37599
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:50:34 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013434, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:34.220041 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.201311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:35135 (local address 127.16.21.62:40035)
0806 10:50:34.201499 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.201504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.202860 (+ 1356us) server_negotiation.cc:407] Connection header received
0806 10:50:34.204799 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.204808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.204904 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.205030 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:34.206486 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.207191 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.208816 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.209275 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.209408 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.212929 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:34.212934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:34.212937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:34.212969 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:34.216856 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:34.217695 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:34.217706 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:34.217709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:34.217851 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:34.218303 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:34.218306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:34.218308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:34.218604 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:34.218766 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.219454 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.219692 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:50:34.221001 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.201861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:35135)
0806 10:50:34.202686 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:34.202730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:34.204530 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:34.205231 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:34.205239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:34.205735 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:34.206293 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.206311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.207368 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.207372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.208652 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.208665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.209516 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.209523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.209751 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.210438 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:34.210467 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:34.212717 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:34.217004 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:34.217013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:34.217027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:34.217557 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:34.217968 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:34.217973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:34.217976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:34.218164 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:34.218738 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:34.218746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:34.219070 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 10:50:34.219323 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 10:50:34.219829 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":362,"threads_started":1}
I20260806 10:50:34.222486 31410 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40035
I20260806 10:50:34.222793 31410 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:34.223631 31410 heartbeater.cc:511] Master 127.16.21.62:40035 requested a full tablet report, sending...
I20260806 10:50:34.227512 31090 ts_manager.cc:194] Registered new tserver with Master: 41a67f43330546adac2b17c4c9d78af0 (127.16.21.2:43471)
I20260806 10:50:34.229434 31090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:35135
W20260806 10:50:34.544746 31413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:34.544983 31413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:34.545071 31413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:34.552194 31413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:34.552335 31413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:34.552479 31413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:50:34.562064 31413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:33993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=37599
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:34.563169 31413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:34.564391 31413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:34.577965 31418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:34.577885 31419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:34.577878 31421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:34.578840 31413 server_base.cc:1034] running on GCE node
I20260806 10:50:34.579432 31413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:34.580413 31413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:34.581631 31413 hybrid_clock.cc:648] HybridClock initialized: now 1786013434581588 us; error 52 us; skew 500 ppm
I20260806 10:50:34.581946 31413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:34 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013434, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:34.585191 31413 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:50:34.586552 31413 webserver.cc:460] Webserver started at http://127.16.21.3:37599/ using document root <none> and password file <none>
I20260806 10:50:34.587205 31413 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:34.587371 31413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:34.592583 31413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:50:34.595238 31428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:34.596639 31413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:34.596827 31413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "2380a6ddb6754395b7f51b3baa326213"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "87d8c3ed74e8503d4e31c12248ab58ff"
server_key_iv: "425a7cb9b0cf41677ba432ca86cc7fe5"
server_key_version: "encryptionkey@0"
I20260806 10:50:34.597831 31413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:34.621583 31413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:34.625460 31413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:34.625732 31413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:34.626703 31413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:34.627990 31413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:34.628119 31413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:34.628299 31413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:34.628399 31413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 10:50:34.662096 31413 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:33993
I20260806 10:50:34.662189 31541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:33993 every 8 connection(s)
I20260806 10:50:34.663751 31413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:50:34.670348 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31413
Aug 06 10:50:34 dist-test-slave-0b58 krb5kdc[30499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013434, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:34.685824 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.666489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:42155 (local address 127.16.21.62:40035)
0806 10:50:34.666664 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.666668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.667758 (+ 1090us) server_negotiation.cc:407] Connection header received
0806 10:50:34.669668 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.669673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.669769 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.669884 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:34.671829 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.673028 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.674823 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.675330 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.675459 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.679447 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:34.679454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:34.679458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:34.679514 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:34.682662 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:34.683440 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:34.683448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:34.683452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:34.683625 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:34.684111 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:34.684116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:34.684119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:34.684461 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:34.684615 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.685269 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.685480 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:50:34.686506 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.671639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.16.21.62:40035)
0806 10:50:34.672155 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.672159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.672172 (+ 13us) server_negotiation.cc:407] Connection header received
0806 10:50:34.672209 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.672211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.672307 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.672402 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:34.674084 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.675108 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.676778 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.677472 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.677621 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.680568 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:34.680573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:34.680576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:34.680618 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:34.683639 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:34.684481 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:34.684490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:34.684492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:34.684609 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:34.685021 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:34.685026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:34.685035 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:34.685287 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:34.685549 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.685874 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.686087 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":332,"threads_started":1}
I20260806 10:50:34.686969 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.666896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:42155)
0806 10:50:34.667592 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:34.667635 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:34.669405 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:34.670126 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:34.670134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:34.670759 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:34.671596 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.671623 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.673184 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.673188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.674677 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.674691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.675547 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.675552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.675741 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.676441 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:34.676468 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:34.679207 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:34.682821 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:34.682830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:34.682844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:34.683318 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:34.683727 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:34.683732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:34.683735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:34.683978 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:34.684591 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:34.684599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:34.684919 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 10:50:34.685178 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 10:50:34.685678 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":282,"threads_started":1}
I20260806 10:50:34.688318 31542 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40035
I20260806 10:50:34.688618 31542 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:34.689320 31542 heartbeater.cc:511] Master 127.16.21.62:40035 requested a full tablet report, sending...
I20260806 10:50:34.690668 31090 ts_manager.cc:194] Registered new tserver with Master: 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993)
I20260806 10:50:34.691481 31090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:42155
I20260806 10:50:34.701341 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:50:34.708033 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.702445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.16.21.62:40035)
0806 10:50:34.702609 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.702615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.703103 (+ 488us) server_negotiation.cc:407] Connection header received
0806 10:50:34.703290 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.703294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.703427 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.703545 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:50:34.703910 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.704669 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.706024 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.706437 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.706525 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.706937 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:50:34.707577 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:50:34.707680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.707792 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.707857 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:50:34.708938 31090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:50:34.716379 31090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:50:34.747460 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.728305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33993 (local address 127.0.0.1:47460)
0806 10:50:34.737344 (+ 9039us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:34.737396 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:34.737601 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:34.740070 (+ 2469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:34.740075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:34.740103 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:34.740638 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.740650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.742613 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.742622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.744589 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.744602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.746409 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.746416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.746616 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.746667 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 10:50:34.746807 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 10:50:34.746992 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8688,"mutex_wait_us":25,"thread_start_us":9640,"threads_started":1}
I20260806 10:50:34.747598 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.729156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.16.21.1:45515)
0806 10:50:34.739131 (+ 9975us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.739142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.739177 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:50:34.739275 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.739282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.739520 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.740539 (+ 1019us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:34.740796 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.742398 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.744981 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.746419 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.746533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.746756 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.747111 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.747279 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9759,"thread_start_us":9525,"threads_started":1}
I20260806 10:50:34.747927 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.728564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:45515 (local address 127.0.0.1:48804)
0806 10:50:34.738197 (+ 9633us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:34.738223 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:34.738353 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:34.739741 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:34.739749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:34.739797 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:34.740275 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.740292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.742552 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.742560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.744796 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.744811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.746778 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.746787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.746925 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.746948 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:50:34.747046 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:50:34.747179 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9508,"mutex_wait_us":97,"thread_start_us":10299,"threads_started":1}
I20260806 10:50:34.748781 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.728555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.16.21.3:33993)
0806 10:50:34.739523 (+ 10968us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.739532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.739560 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:50:34.739648 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.739654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.739857 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.740843 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:34.741135 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.742488 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.744744 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.746198 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.746351 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.746597 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.748208 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.748417 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10805,"thread_start_us":10653,"threads_started":1}
I20260806 10:50:34.748863 31554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.728977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:43471 (local address 127.0.0.1:34606)
0806 10:50:34.739016 (+ 10039us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:34.739034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:34.739251 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:34.741235 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:34.741241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:34.741280 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:34.741641 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.741652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.743586 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.743591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.745901 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:34.745914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.748073 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.748080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:34.748243 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.748310 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 10:50:34.748392 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 10:50:34.748509 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9921,"mutex_wait_us":56,"thread_start_us":7842,"threads_started":1}
I20260806 10:50:34.749189 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:34.729559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.16.21.2:43471)
0806 10:50:34.730293 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:34.730307 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:34.739139 (+ 8832us) server_negotiation.cc:407] Connection header received
0806 10:50:34.739460 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:34.739467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:34.739714 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:34.739872 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:34.741887 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.743439 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.746085 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:34.747521 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:34.747661 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:34.747871 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:34.748478 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 10:50:34.748647 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":332,"threads_started":1}
I20260806 10:50:34.752425 31476 tablet_service.cc:1467] Processing CreateTablet for tablet 9621191364274c14a91cfb252842b805 (DEFAULT_TABLE table=test-table [id=f81a18a3234e4fe0b586ace2031bdb0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:34.752246 31211 tablet_service.cc:1467] Processing CreateTablet for tablet 9621191364274c14a91cfb252842b805 (DEFAULT_TABLE table=test-table [id=f81a18a3234e4fe0b586ace2031bdb0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:34.752790 31345 tablet_service.cc:1467] Processing CreateTablet for tablet 9621191364274c14a91cfb252842b805 (DEFAULT_TABLE table=test-table [id=f81a18a3234e4fe0b586ace2031bdb0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:34.753697 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9621191364274c14a91cfb252842b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:34.753791 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9621191364274c14a91cfb252842b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:34.754022 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9621191364274c14a91cfb252842b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:34.762964 31277 heartbeater.cc:503] Master 127.16.21.62:40035 was elected leader, sending a full tablet report...
I20260806 10:50:34.768364 31558 tablet_bootstrap.cc:492] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Bootstrap starting.
I20260806 10:50:34.769522 31559 tablet_bootstrap.cc:492] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Bootstrap starting.
I20260806 10:50:34.770390 31560 tablet_bootstrap.cc:492] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Bootstrap starting.
I20260806 10:50:34.772773 31558 tablet_bootstrap.cc:654] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:34.773983 31560 tablet_bootstrap.cc:654] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:34.774116 31558 log.cc:824] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:34.775116 31559 tablet_bootstrap.cc:654] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:34.775166 31560 log.cc:824] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:34.776408 31559 log.cc:824] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:34.777426 31558 tablet_bootstrap.cc:492] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: No bootstrap required, opened a new log
I20260806 10:50:34.778211 31558 ts_tablet_manager.cc:1397] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 10:50:34.778337 31560 tablet_bootstrap.cc:492] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: No bootstrap required, opened a new log
I20260806 10:50:34.778633 31560 ts_tablet_manager.cc:1397] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 10:50:34.779147 31559 tablet_bootstrap.cc:492] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: No bootstrap required, opened a new log
I20260806 10:50:34.779414 31559 ts_tablet_manager.cc:1397] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 10:50:34.786612 31559 raft_consensus.cc:348] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:34.787017 31559 raft_consensus.cc:374] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:34.787125 31559 raft_consensus.cc:729] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a67f43330546adac2b17c4c9d78af0, State: Initialized, Role: FOLLOWER
I20260806 10:50:34.787727 31559 consensus_queue.cc:260] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:34.788852 31410 heartbeater.cc:503] Master 127.16.21.62:40035 was elected leader, sending a full tablet report...
I20260806 10:50:34.789464 31559 ts_tablet_manager.cc:1428] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260806 10:50:34.789529 31558 raft_consensus.cc:348] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:34.789531 31560 raft_consensus.cc:348] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:34.789933 31560 raft_consensus.cc:374] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:34.789921 31558 raft_consensus.cc:374] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:34.790026 31560 raft_consensus.cc:729] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4c4f051d624ad1bed97dce3098b190, State: Initialized, Role: FOLLOWER
I20260806 10:50:34.790020 31558 raft_consensus.cc:729] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2380a6ddb6754395b7f51b3baa326213, State: Initialized, Role: FOLLOWER
I20260806 10:50:34.790673 31558 consensus_queue.cc:260] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:34.790652 31560 consensus_queue.cc:260] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:34.791829 31542 heartbeater.cc:503] Master 127.16.21.62:40035 was elected leader, sending a full tablet report...
I20260806 10:50:34.792667 31558 ts_tablet_manager.cc:1428] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 10:50:34.794636 31560 ts_tablet_manager.cc:1428] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Time spent starting tablet: real 0.016s user 0.006s sys 0.008s
W20260806 10:50:34.915714 31543 tablet.cc:2369] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:50:34.950575 31411 tablet.cc:2369] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:50:34.979863 31278 tablet.cc:2369] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:35.111747 31564 raft_consensus.cc:482] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:50:35.112062 31564 raft_consensus.cc:504] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:35.113902 31564 leader_election.cc:288] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993), ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515)
I20260806 10:50:35.120579 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.114328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:49297 (local address 127.16.21.3:33993)
0806 10:50:35.114531 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:35.114538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:35.115303 (+ 765us) server_negotiation.cc:407] Connection header received
0806 10:50:35.115487 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:35.115497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:35.115554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:35.115646 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:35.116220 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.116995 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.118872 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.119620 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.119769 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.119882 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:35.120168 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 10:50:35.120321 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:50:35.120596 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.114546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:39095 (local address 127.16.21.1:45515)
0806 10:50:35.114732 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:35.114737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:35.115374 (+ 637us) server_negotiation.cc:407] Connection header received
0806 10:50:35.115569 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:35.115578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:35.115637 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:35.115731 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:35.116450 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.117155 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.118888 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.119635 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.119796 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.119911 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:35.120209 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 10:50:35.120349 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:50:35.120780 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.114307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33993 (local address 127.16.21.2:49297)
0806 10:50:35.115179 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:35.115200 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:35.115336 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:35.115668 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:35.115672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:35.115697 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:35.116081 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:35.116090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.117122 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.117128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:35.118716 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:35.118728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.119864 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.119872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:35.120011 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.120037 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:50:35.120142 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 10:50:35.120320 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":68,"spinlock_wait_cycles":28928,"thread_start_us":688,"threads_started":1}
I20260806 10:50:35.120824 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.114461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:45515 (local address 127.16.21.2:39095)
0806 10:50:35.115250 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:35.115294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:35.115433 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:35.115746 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:35.115754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:35.115856 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:35.116232 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:35.116241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.117302 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.117307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:35.118735 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:35.118746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.119903 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.119910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:35.120058 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.120080 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 10:50:35.120183 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 10:50:35.120324 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":21,"spinlock_wait_cycles":32640,"thread_start_us":383,"threads_started":1}
I20260806 10:50:35.121905 31496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621191364274c14a91cfb252842b805" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2380a6ddb6754395b7f51b3baa326213" is_pre_election: true
I20260806 10:50:35.121907 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621191364274c14a91cfb252842b805" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4c4f051d624ad1bed97dce3098b190" is_pre_election: true
I20260806 10:50:35.122403 31496 raft_consensus.cc:2454] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 0.
I20260806 10:50:35.122411 31231 raft_consensus.cc:2454] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 0.
I20260806 10:50:35.123020 31300 leader_election.cc:302] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a67f43330546adac2b17c4c9d78af0, ee4c4f051d624ad1bed97dce3098b190; no voters:
I20260806 10:50:35.123407 31564 raft_consensus.cc:2781] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:50:35.123569 31564 raft_consensus.cc:482] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:50:35.123685 31564 raft_consensus.cc:3037] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:35.125993 31564 raft_consensus.cc:504] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:35.126650 31564 leader_election.cc:288] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 election: Requested vote from peers 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993), ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515)
I20260806 10:50:35.127008 31496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621191364274c14a91cfb252842b805" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2380a6ddb6754395b7f51b3baa326213"
I20260806 10:50:35.127209 31496 raft_consensus.cc:3037] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:35.127308 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621191364274c14a91cfb252842b805" candidate_uuid: "41a67f43330546adac2b17c4c9d78af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4c4f051d624ad1bed97dce3098b190"
I20260806 10:50:35.127542 31231 raft_consensus.cc:3037] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:35.129558 31496 raft_consensus.cc:2454] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 1.
I20260806 10:50:35.129820 31231 raft_consensus.cc:2454] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a67f43330546adac2b17c4c9d78af0 in term 1.
I20260806 10:50:35.129940 31297 leader_election.cc:302] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2380a6ddb6754395b7f51b3baa326213, 41a67f43330546adac2b17c4c9d78af0; no voters:
I20260806 10:50:35.130257 31564 raft_consensus.cc:2781] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:35.130645 31564 raft_consensus.cc:686] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 1 LEADER]: Becoming Leader. State: Replica: 41a67f43330546adac2b17c4c9d78af0, State: Running, Role: LEADER
I20260806 10:50:35.131141 31564 consensus_queue.cc:237] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } }
I20260806 10:50:35.135524 31090 catalog_manager.cc:5495] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41a67f43330546adac2b17c4c9d78af0 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "41a67f43330546adac2b17c4c9d78af0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a67f43330546adac2b17c4c9d78af0" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 43471 } health_report { overall_health: HEALTHY } } }
I20260806 10:50:35.191193 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.185734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.16.21.2:43471)
0806 10:50:35.185947 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:35.185955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:35.185974 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:50:35.186254 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:35.186258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:35.186347 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:35.186501 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:50:35.186928 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.187647 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.189081 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.189473 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.189609 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.190076 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:50:35.190560 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:50:35.190731 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:35.190852 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 10:50:35.190937 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:50:35.200032 31231 raft_consensus.cc:1261] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Refusing update from remote peer 41a67f43330546adac2b17c4c9d78af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:35.200029 31496 raft_consensus.cc:1261] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Refusing update from remote peer 41a67f43330546adac2b17c4c9d78af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:35.201176 31564 consensus_queue.cc:1035] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4c4f051d624ad1bed97dce3098b190" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:35.201860 31569 consensus_queue.cc:1035] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2380a6ddb6754395b7f51b3baa326213" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:35.230234 31573 mvcc.cc:205] Tried to move back new op lower bound from 7315511030568112128 to 7315511030300999680. Current Snapshot: MvccSnapshot[applied={T|T < 7315511030568112128}]
I20260806 10:50:35.238288 31090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:35.238575 31090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:35.242856 31090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Sending DeleteTablet for 3 replicas of tablet 9621191364274c14a91cfb252842b805
I20260806 10:50:35.244618 31345 tablet_service.cc:1514] Processing DeleteTablet for tablet 9621191364274c14a91cfb252842b805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:35 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:34606
I20260806 10:50:35.244689 31476 tablet_service.cc:1514] Processing DeleteTablet for tablet 9621191364274c14a91cfb252842b805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:35 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:47460
I20260806 10:50:35.245082 31211 tablet_service.cc:1514] Processing DeleteTablet for tablet 9621191364274c14a91cfb252842b805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:35 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:48804
I20260806 10:50:35.246174 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31127
I20260806 10:50:35.279210 31582 tablet_replica.cc:330] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: stopping tablet replica
I20260806 10:50:35.279870 31582 raft_consensus.cc:2229] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:35.280365 31581 tablet_replica.cc:330] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: stopping tablet replica
I20260806 10:50:35.280599 31583 tablet_replica.cc:330] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: stopping tablet replica
I20260806 10:50:35.280856 31581 raft_consensus.cc:2229] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:50:35.281002 31583 raft_consensus.cc:2229] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:35.281214 31582 raft_consensus.cc:2258] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:35.281491 31583 raft_consensus.cc:2258] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:35.281860 31581 raft_consensus.cc:2258] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:35.283607 31583 ts_tablet_manager.cc:1905] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:35.283761 31582 ts_tablet_manager.cc:1905] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:35.285637 31581 ts_tablet_manager.cc:1905] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:35.291100 31583 ts_tablet_manager.cc:1918] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:35.291285 31583 log.cc:1196] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/9621191364274c14a91cfb252842b805
I20260806 10:50:35.291867 31583 ts_tablet_manager.cc:1939] T 9621191364274c14a91cfb252842b805 P ee4c4f051d624ad1bed97dce3098b190: Deleting consensus metadata
I20260806 10:50:35.291894 31582 ts_tablet_manager.cc:1918] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:35.292213 31582 log.cc:1196] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/9621191364274c14a91cfb252842b805
I20260806 10:50:35.292879 31582 ts_tablet_manager.cc:1939] T 9621191364274c14a91cfb252842b805 P 2380a6ddb6754395b7f51b3baa326213: Deleting consensus metadata
I20260806 10:50:35.293087 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.247572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.16.21.1:45515)
0806 10:50:35.247803 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:35.247813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:35.270259 (+ 22446us) server_negotiation.cc:407] Connection header received
0806 10:50:35.270942 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:35.270952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:35.271105 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:35.271202 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:35.274311 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.275940 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.280113 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.280821 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.280943 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.284961 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:35.284976 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:35.284993 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:35.285039 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:35.289189 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:35.290133 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:35.290142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:35.290147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:35.290320 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:35.290816 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:35.290821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:35.290825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:35.291280 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:35.291623 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:35.292347 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 10:50:35.292610 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:50:35.293853 31077 catalog_manager.cc:4841] TS ee4c4f051d624ad1bed97dce3098b190 (127.16.21.1:45515): tablet 9621191364274c14a91cfb252842b805 (table test-table [id=f81a18a3234e4fe0b586ace2031bdb0c]) successfully deleted
I20260806 10:50:35.294497 31272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:35.299006 31581 ts_tablet_manager.cc:1918] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:35.299182 31581 log.cc:1196] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/9621191364274c14a91cfb252842b805
I20260806 10:50:35.299558 31074 catalog_manager.cc:4841] TS 2380a6ddb6754395b7f51b3baa326213 (127.16.21.3:33993): tablet 9621191364274c14a91cfb252842b805 (table test-table [id=f81a18a3234e4fe0b586ace2031bdb0c]) successfully deleted
I20260806 10:50:35.299762 31581 ts_tablet_manager.cc:1939] T 9621191364274c14a91cfb252842b805 P 41a67f43330546adac2b17c4c9d78af0: Deleting consensus metadata
I20260806 10:50:35.301540 31076 catalog_manager.cc:4841] TS 41a67f43330546adac2b17c4c9d78af0 (127.16.21.2:43471): tablet 9621191364274c14a91cfb252842b805 (table test-table [id=f81a18a3234e4fe0b586ace2031bdb0c]) successfully deleted
I20260806 10:50:35.385733 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31127
I20260806 10:50:35.407418 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31281
I20260806 10:50:35.423058 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.408472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.16.21.2:43471)
0806 10:50:35.408685 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:35.408692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:35.408714 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:50:35.408787 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:35.408793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:35.408872 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:35.408990 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:35.410331 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.411638 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.412997 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.413671 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.413805 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.416196 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:35.416211 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:35.416230 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:35.416288 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:35.420207 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:35.420925 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:35.420933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:35.420937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:35.421104 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:35.421490 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:35.421495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:35.421499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:35.421933 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:35.422277 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:35.422432 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 10:50:35.422666 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:50:35.423875 31405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:35.539139 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31281
I20260806 10:50:35.561256 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31413
I20260806 10:50:35.579911 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:35.562038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.16.21.3:33993)
0806 10:50:35.562221 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:35.562227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:35.562306 (+ 79us) server_negotiation.cc:407] Connection header received
0806 10:50:35.562538 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:35.562542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:35.562614 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:35.562722 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:35.564116 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.564901 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.566403 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:35.566839 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:35.566999 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:35.574282 (+ 7283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:35.574299 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:35.574314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:35.574348 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:35.577045 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:35.577774 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:35.577790 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:35.577793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:35.577926 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:35.578316 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:35.578323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:35.578326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:35.578709 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:35.579046 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:35.579340 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 10:50:35.579560 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:50:35.580675 31537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:35.671155 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31413
I20260806 10:50:35.691043 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31057
I20260806 10:50:35.692374 31120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:35.807808 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31057
I20260806 10:50:35.833660 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40035
--webserver_interface=127.16.21.62
--webserver_port=41065
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:36.193099 31589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:36.193387 31589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:36.193480 31589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:36.200644 31589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:36.200742 31589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:36.200800 31589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:36.200858 31589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:36.200903 31589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:36.210410 31589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=41065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:36.211586 31589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:36.212788 31589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:36.224493 31597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:36.224603 31594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:36.224565 31595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:36.225893 31589 server_base.cc:1034] running on GCE node
I20260806 10:50:36.227038 31589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:36.229128 31589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:36.230388 31589 hybrid_clock.cc:648] HybridClock initialized: now 1786013436230315 us; error 61 us; skew 500 ppm
I20260806 10:50:36.230844 31589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:36 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:36.234606 31589 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:36.236048 31589 webserver.cc:460] Webserver started at http://127.16.21.62:41065/ using document root <none> and password file <none>
I20260806 10:50:36.236819 31589 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:36.236974 31589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:36.242412 31589 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 10:50:36.245223 31604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:36.246541 31589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:50:36.246728 31589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "5d4c44064a2e42a2a138bbea48f10fb7"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "920ed875fbdf75c1328f1e9d1413c004"
server_key_iv: "0125b164f2c7eb1b4a2b56e7846e695b"
server_key_version: "encryptionkey@0"
I20260806 10:50:36.247705 31589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:36.290241 31589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:36.294415 31589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:36.294690 31589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:36.314740 31589 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40035
I20260806 10:50:36.314782 31655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40035 every 8 connection(s)
I20260806 10:50:36.316462 31589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:36.323221 31656 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:50:36.323601 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31589
I20260806 10:50:36.324519 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:45515
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:40035
--webserver_port=41065
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:36.326784 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap starting.
I20260806 10:50:36.351197 31656 log.cc:824] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:36.373780 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:50:36.374390 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7: Bootstrap complete.
I20260806 10:50:36.387854 31656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:36.389226 31656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c44064a2e42a2a138bbea48f10fb7, State: Initialized, Role: FOLLOWER
I20260806 10:50:36.390038 31656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:36.390357 31656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:36.390499 31656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:36.390714 31656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 10:50:36.396961 31656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:36.397555 31656 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4c44064a2e42a2a138bbea48f10fb7; no voters:
I20260806 10:50:36.398247 31656 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 10:50:36.398425 31660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 10:50:36.399711 31660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [term 3 LEADER]: Becoming Leader. State: Replica: 5d4c44064a2e42a2a138bbea48f10fb7, State: Running, Role: LEADER
I20260806 10:50:36.400347 31660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } }
I20260806 10:50:36.400635 31656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:36.402680 31662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4c44064a2e42a2a138bbea48f10fb7. Latest consensus state: current_term: 3 leader_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } } }
I20260806 10:50:36.403062 31662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:36.411206 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4c44064a2e42a2a138bbea48f10fb7" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40035 } } }
I20260806 10:50:36.411487 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4c44064a2e42a2a138bbea48f10fb7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:36.411532 31666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:36.420676 31666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:36.422130 31666 catalog_manager.cc:1259] Loaded cluster ID: 759e700790204cfd9c2af943ea3ec599
I20260806 10:50:36.422232 31666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 10:50:36.424307 31666 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f06dd54e9ad google::LogMessage::Fail() at ??:0
@ 0x7f06dd55188a google::LogMessage::SendToLog() at ??:0
@ 0x7f06dd54e4de google::LogMessage::Flush() at ??:0
@ 0x7f06dd54fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f06f2f81512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f06f2f7e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f06f3006610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f06f3006439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f06e019a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f06e019d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f06e019d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f06e0178eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f06e34526db start_thread at ??:0
@ 0x7f06dc3e861f clone at ??:0
W20260806 10:50:36.739813 31658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:36.740111 31658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:36.740309 31658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:36.747493 31658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:36.747635 31658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:36.747822 31658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:50:36.757661 31658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:45515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=41065
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:36.758886 31658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:36.760241 31658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:36.773380 31698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:36.773416 31701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:36.773468 31699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:36.774525 31658 server_base.cc:1034] running on GCE node
I20260806 10:50:36.775071 31658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:36.775906 31658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:36.777098 31658 hybrid_clock.cc:648] HybridClock initialized: now 1786013436777047 us; error 44 us; skew 500 ppm
I20260806 10:50:36.777388 31658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:36 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013436, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:36.780592 31658 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:50:36.782096 31658 webserver.cc:460] Webserver started at http://127.16.21.1:41065/ using document root <none> and password file <none>
I20260806 10:50:36.782794 31658 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:36.782946 31658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:36.788151 31658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:50:36.790783 31708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:36.792048 31658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:50:36.792230 31658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "ee4c4f051d624ad1bed97dce3098b190"
format_stamp: "Formatted at 2026-08-06 10:50:30 on dist-test-slave-0b58"
server_key: "d3c8d607afe9226f12a0755ab31b6012"
server_key_iv: "890110ce0b8db3f0636f1c3eb2b74fca"
server_key_version: "encryptionkey@0"
I20260806 10:50:36.793298 31658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:36.815842 31658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:36.819671 31658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:36.819952 31658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:36.820948 31658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:36.822400 31658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:36.822532 31658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:36.822661 31658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:36.822750 31658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:36.857354 31658 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:45515
I20260806 10:50:36.857436 31820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:45515 every 8 connection(s)
I20260806 10:50:36.858762 31658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:50:36.861620 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31658
I20260806 10:50:36.862421 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:43471
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:40035
--webserver_port=39381
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:50:36.864027 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:36.861692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:48961)
0806 10:50:36.862602 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:36.862865 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":319,"threads_started":1}
W20260806 10:50:36.864811 31821 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:40035 (0 consecutive failures): Network error: Failed to ping master at 127.16.21.62:40035: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
I20260806 10:50:36.865804 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:36.865388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:44911)
0806 10:50:36.865568 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:36.865658 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 10:50:36.866712 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:36.866397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:48973)
0806 10:50:36.866526 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:36.866601 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:50:36.867002 31821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:50:37.200179 31824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:37.200605 31824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:37.200799 31824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:37.207942 31824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:37.208065 31824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:37.208269 31824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:50:37.217422 31824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:43471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=39381
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:37.218595 31824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:37.219830 31824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:37.233311 31829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:37.233215 31832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:37.233341 31830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:37.234190 31824 server_base.cc:1034] running on GCE node
I20260806 10:50:37.234844 31824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:37.235762 31824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:37.236996 31824 hybrid_clock.cc:648] HybridClock initialized: now 1786013437236958 us; error 62 us; skew 500 ppm
I20260806 10:50:37.237320 31824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:37 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013437, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:37.240407 31824 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:50:37.241741 31824 webserver.cc:460] Webserver started at http://127.16.21.2:39381/ using document root <none> and password file <none>
I20260806 10:50:37.242419 31824 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:37.242566 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:37.247684 31824 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:50:37.250336 31839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:37.251523 31824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:50:37.251698 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "41a67f43330546adac2b17c4c9d78af0"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "8875c3b2745c7030b935910a7bfa4714"
server_key_iv: "cc45b7b9313e35a5b0d140e053cc5f7c"
server_key_version: "encryptionkey@0"
I20260806 10:50:37.252743 31824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:37.289863 31824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:37.294142 31824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:37.294387 31824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:37.295303 31824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:37.296718 31824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:37.296838 31824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:37.296921 31824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:37.296978 31824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:37.331252 31824 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:43471
I20260806 10:50:37.331352 31951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:43471 every 8 connection(s)
I20260806 10:50:37.332770 31824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:50:37.337519 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.335691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:53517)
0806 10:50:37.336503 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.336702 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":296,"threads_started":1}
W20260806 10:50:37.338201 31952 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:40035 (0 consecutive failures): Network error: Failed to ping master at 127.16.21.62:40035: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
I20260806 10:50:37.338940 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.338632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:43559)
0806 10:50:37.338769 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.338844 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:50:37.339763 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.339426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:52651)
0806 10:50:37.339580 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.339659 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 10:50:37.340013 31952 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:50:37.343051 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31824
I20260806 10:50:37.343658 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:33993
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:40035
--webserver_port=37599
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:37.680787 31955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:37.681047 31955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:37.681201 31955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:37.688367 31955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:37.688482 31955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:37.688633 31955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:50:37.698253 31955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:33993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=37599
--tserver_master_addrs=127.16.21.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:37.699471 31955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:37.700809 31955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:37.716634 31960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:37.716722 31963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:37.716744 31961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:37.717527 31955 server_base.cc:1034] running on GCE node
I20260806 10:50:37.718123 31955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:37.719051 31955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:37.720314 31955 hybrid_clock.cc:648] HybridClock initialized: now 1786013437720231 us; error 52 us; skew 500 ppm
I20260806 10:50:37.720647 31955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:37 dist-test-slave-0b58 krb5kdc[30499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013437, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:37.723685 31955 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:50:37.725152 31955 webserver.cc:460] Webserver started at http://127.16.21.3:37599/ using document root <none> and password file <none>
I20260806 10:50:37.725823 31955 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:37.725975 31955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:37.731155 31955 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 10:50:37.733879 31970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:37.735148 31955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:50:37.735328 31955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "2380a6ddb6754395b7f51b3baa326213"
format_stamp: "Formatted at 2026-08-06 10:50:31 on dist-test-slave-0b58"
server_key: "87d8c3ed74e8503d4e31c12248ab58ff"
server_key_iv: "425a7cb9b0cf41677ba432ca86cc7fe5"
server_key_version: "encryptionkey@0"
I20260806 10:50:37.736321 31955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:37.787390 31955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:37.791608 31955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:37.791888 31955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:37.792969 31955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:37.794344 31955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:37.794480 31955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:37.794613 31955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:37.794708 31955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:37.829625 31955 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:33993
I20260806 10:50:37.829677 32082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:33993 every 8 connection(s)
I20260806 10:50:37.831171 31955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:50:37.833926 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 31955
I20260806 10:50:37.836220 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.834028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:60517)
0806 10:50:37.834902 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.835111 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":359,"threads_started":1}
W20260806 10:50:37.836921 32083 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:40035 (0 consecutive failures): Network error: Failed to ping master at 127.16.21.62:40035: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
I20260806 10:50:37.837630 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.837334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:57233)
0806 10:50:37.837468 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.837531 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:50:37.838477 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.838146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:48773)
0806 10:50:37.838294 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.838373 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 10:50:37.838734 32083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:50:37.868222 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:37.867459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:40031)
0806 10:50:37.867886 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:37.867976 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":206,"threads_started":1}
I20260806 10:50:38.341302 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:38.340505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:34261)
0806 10:50:38.340967 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:38.341077 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260806 10:50:38.840020 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:38.839250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:60221)
0806 10:50:38.839706 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:38.839797 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":233,"threads_started":1}
I20260806 10:50:38.869777 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:38.869093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:35049)
0806 10:50:38.869535 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:38.869611 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":224,"threads_started":1}
I20260806 10:50:39.342721 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:39.342090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:41269)
0806 10:50:39.342488 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:39.342564 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":205,"threads_started":1}
I20260806 10:50:39.841517 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:39.840881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:55433)
0806 10:50:39.841279 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:39.841361 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":208,"threads_started":1}
I20260806 10:50:39.871328 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:39.870626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:59133)
0806 10:50:39.871038 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:39.871136 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260806 10:50:40.344375 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:40.343599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:33933)
0806 10:50:40.344039 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:40.344135 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260806 10:50:40.842991 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:40.842311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:52383)
0806 10:50:40.842748 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:40.842825 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260806 10:50:40.873193 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:40.872351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:34259)
0806 10:50:40.872928 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:40.873003 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":370,"threads_started":1}
I20260806 10:50:41.345836 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:41.345163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:48745)
0806 10:50:41.345587 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:41.345662 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":217,"threads_started":1}
I20260806 10:50:41.844576 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:41.843773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:45929)
0806 10:50:41.844334 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:41.844410 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":299,"threads_started":1}
I20260806 10:50:41.874763 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:41.874076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:58139)
0806 10:50:41.874531 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:41.874604 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260806 10:50:42.347277 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:42.346618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:48621)
0806 10:50:42.347040 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:42.347118 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260806 10:50:42.846007 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:42.845338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:55723)
0806 10:50:42.845773 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:42.845851 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260806 10:50:42.876372 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:42.875587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:38923)
0806 10:50:42.876056 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:42.876138 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260806 10:50:43.349000 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:43.348135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:53569)
0806 10:50:43.348749 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:43.348839 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":418,"threads_started":1}
I20260806 10:50:43.847540 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:43.846835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:60071)
0806 10:50:43.847277 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:43.847368 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260806 10:50:43.878232 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:43.877313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:36319)
0806 10:50:43.877910 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:43.877996 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":295,"threads_started":1}
I20260806 10:50:44.350713 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:44.349894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:38411)
0806 10:50:44.350419 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:44.350502 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":228,"threads_started":1}
I20260806 10:50:44.849295 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:44.848501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:39537)
0806 10:50:44.848986 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:44.849074 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":217,"threads_started":1}
I20260806 10:50:44.879822 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:44.879080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:35855)
0806 10:50:44.879526 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:44.879628 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260806 10:50:45.352222 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:45.351576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:43491)
0806 10:50:45.351985 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:45.352063 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":222,"threads_started":1}
I20260806 10:50:45.850879 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:45.850171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:35133)
0806 10:50:45.850625 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:45.850705 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260806 10:50:45.881390 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:45.880655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:43449)
0806 10:50:45.881125 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:45.881202 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260806 10:50:46.353739 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:46.353052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:37133)
0806 10:50:46.353493 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:46.353576 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260806 10:50:46.852468 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:46.851655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:59891)
0806 10:50:46.852115 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:46.852191 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260806 10:50:46.883109 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:46.882259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:41261)
0806 10:50:46.882796 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:46.882892 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":297,"threads_started":1}
I20260806 10:50:47.355242 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:47.354575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:35797)
0806 10:50:47.355010 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:47.355085 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":252,"threads_started":1}
I20260806 10:50:47.853935 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:47.853251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:60785)
0806 10:50:47.853700 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:47.853777 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260806 10:50:47.884666 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:47.883939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:59247)
0806 10:50:47.884401 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:47.884495 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":198,"threads_started":1}
I20260806 10:50:48.356783 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:48.356052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:44867)
0806 10:50:48.356516 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:48.356605 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":271,"threads_started":1}
I20260806 10:50:48.855398 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:48.854730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:52753)
0806 10:50:48.855136 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:48.855219 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":198,"threads_started":1}
I20260806 10:50:48.886265 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:48.885602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:35273)
0806 10:50:48.886017 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:48.886091 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":219,"threads_started":1}
I20260806 10:50:49.358211 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:49.357577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:59991)
0806 10:50:49.357984 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:49.358061 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":215,"threads_started":1}
I20260806 10:50:49.857326 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:49.856453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:42061)
0806 10:50:49.857004 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:49.857106 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":246,"threads_started":1}
I20260806 10:50:49.887825 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:49.887095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:43881)
0806 10:50:49.887539 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:49.887622 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":213,"threads_started":1}
I20260806 10:50:50.359915 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:50.359032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:39121)
0806 10:50:50.359566 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:50.359678 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
I20260806 10:50:50.858976 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:50.858137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:33693)
0806 10:50:50.858657 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:50.858768 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":269,"threads_started":1}
I20260806 10:50:50.889477 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:50.888741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:35137)
0806 10:50:50.889208 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:50.889288 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":252,"threads_started":1}
I20260806 10:50:51.361655 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:51.360888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:48897)
0806 10:50:51.361390 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:51.361476 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":277,"threads_started":1}
I20260806 10:50:51.860585 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:51.859788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:51813)
0806 10:50:51.860324 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:51.860403 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":297,"threads_started":1}
I20260806 10:50:51.891477 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:51.890598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:46687)
0806 10:50:51.891155 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:51.891274 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":334,"threads_started":1}
I20260806 10:50:52.363811 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:52.362795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.2:57747)
0806 10:50:52.363457 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:52.363554 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":416,"threads_started":1}
I20260806 10:50:52.845487 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31658
I20260806 10:50:52.862429 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:52.861661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.3:53393)
0806 10:50:52.862158 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:52.862270 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":283,"threads_started":1}
I20260806 10:50:52.868813 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:52.846549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.16.21.1:45515)
0806 10:50:52.847234 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:52.847252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:52.847296 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:50:52.849698 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:52.849708 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:52.850125 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:52.850418 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:52.851755 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:52.853074 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:52.854463 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:52.855110 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:52.855248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:52.862963 (+ 7715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:52.862971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:52.862986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:52.863021 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:52.865940 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:52.866572 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:52.866579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:52.866586 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:52.866751 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:52.867082 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:52.867085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:52.867087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:52.867576 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:52.867795 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:52.868158 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 10:50:52.868547 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":339,"threads_started":1}
I20260806 10:50:52.870091 31816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:52.954499 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:52.953528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40035 (local address 127.16.21.1:39403)
0806 10:50:52.954046 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:52.954223 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":238,"threads_started":1}
I20260806 10:50:52.954533 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31658
I20260806 10:50:52.976770 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31824
I20260806 10:50:52.993944 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:52.977671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.16.21.2:43471)
0806 10:50:52.978255 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:52.978269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:52.978306 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:50:52.979901 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:52.979911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:52.980401 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:52.980709 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:52.982115 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:52.983363 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:52.984781 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:52.985423 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:52.985553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:52.988159 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:52.988166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:52.988179 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:52.988221 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:52.991102 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:52.991725 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:52.991731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:52.991737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:52.991911 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:52.992217 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:52.992221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:52.992223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:52.992765 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:52.993014 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:52.993341 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 10:50:52.993648 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":306,"threads_started":1}
I20260806 10:50:52.995293 31947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:53.080500 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31824
I20260806 10:50:53.101269 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 31955
I20260806 10:50:53.118484 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:53.102213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.16.21.3:33993)
0806 10:50:53.102811 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:53.102825 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:53.102866 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:50:53.104570 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:53.104580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:53.104993 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:53.105283 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:53.106582 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:53.107791 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:53.109103 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:53.109737 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:53.109869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:53.112489 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:53.112496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:53.112510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:53.112544 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:53.115539 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:53.116112 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:53.116118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:53.116124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:53.116418 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:53.116737 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:53.116740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:53.116742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:53.117217 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:53.117470 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:53.117781 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 10:50:53.118178 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":279,"threads_started":1}
I20260806 10:50:53.119792 32078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:53.203235 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 31955
2026-08-06T10:50:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24965 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:50:53 dist-test-slave-0b58 krb5kdc[32141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:50:53 dist-test-slave-0b58 krb5kdc[32141](info): set up 2 sockets
Aug 06 10:50:53 dist-test-slave-0b58 krb5kdc[32141](info): commencing operation
krb5kdc: starting...
W20260806 10:50:54.336177 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:50:54 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:50:54.607753 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40767
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:54.939064 32154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:54.939339 32154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:54.939426 32154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:54.946489 32154 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 10:50:54.946575 32154 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 10:50:54.946612 32154 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 10:50:54.946664 32154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:50:54.946718 32154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:54.946789 32154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:54.946833 32154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:50:54.946880 32154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:50:54.956161 32154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:54.957273 32154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:54.958484 32154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:54.970235 32159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:54.970321 32162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:54.970366 32160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:54.970679 32154 server_base.cc:1034] running on GCE node
I20260806 10:50:54.971501 32154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:54.973320 32154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:54.974603 32154 hybrid_clock.cc:648] HybridClock initialized: now 1786013454974527 us; error 43 us; skew 500 ppm
I20260806 10:50:54.974936 32154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:54 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013454, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:54.978091 32154 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:50:54.979581 32154 webserver.cc:460] Webserver started at http://127.16.21.62:39039/ using document root <none> and password file <none>
I20260806 10:50:54.980247 32154 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:54.980448 32154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:54.980778 32154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:54.983716 32154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "bbe6995a967140c08251a12b812e7f01"
format_stamp: "Formatted at 2026-08-06 10:50:54 on dist-test-slave-0b58"
server_key: "932a111ff47f0d8165a1e3785d3075ef"
server_key_iv: "b70b3e170bf5b40f9608d65d5cc40925"
server_key_version: "encryptionkey@0"
I20260806 10:50:54.984552 32154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "bbe6995a967140c08251a12b812e7f01"
format_stamp: "Formatted at 2026-08-06 10:50:54 on dist-test-slave-0b58"
server_key: "932a111ff47f0d8165a1e3785d3075ef"
server_key_iv: "b70b3e170bf5b40f9608d65d5cc40925"
server_key_version: "encryptionkey@0"
I20260806 10:50:54.989507 32154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:50:54.993072 32169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:54.994259 32154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:50:54.994451 32154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "bbe6995a967140c08251a12b812e7f01"
format_stamp: "Formatted at 2026-08-06 10:50:54 on dist-test-slave-0b58"
server_key: "932a111ff47f0d8165a1e3785d3075ef"
server_key_iv: "b70b3e170bf5b40f9608d65d5cc40925"
server_key_version: "encryptionkey@0"
I20260806 10:50:54.994680 32154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:55.012224 32154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:55.016059 32154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:55.016402 32154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:55.035444 32154 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40767
I20260806 10:50:55.035465 32220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40767 every 8 connection(s)
I20260806 10:50:55.036842 32154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:50:55.040412 32221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:55.043147 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 32154
I20260806 10:50:55.043378 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:50:55.044324 16468 external_mini_cluster.cc:1428] Setting key b9003b35de5527ab4f8bc952771a5fc5
I20260806 10:50:55.053113 32221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap starting.
I20260806 10:50:55.056824 32221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:55.058138 32221 log.cc:824] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:50:55 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:55.060977 32221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: No bootstrap required, opened a new log
I20260806 10:50:55.067881 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:55.046368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.16.21.62:40767)
0806 10:50:55.047105 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:55.047116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:55.047156 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:50:55.048853 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:55.048877 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:55.049232 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:55.049547 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:55.051493 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:55.052832 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:55.054769 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:55.055411 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:55.055531 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:55.060587 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:55.060594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:55.060606 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:55.060637 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:55.063731 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:55.064537 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:55.064545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:55.064558 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:55.064715 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:55.065157 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:55.065163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:55.065168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:55.065663 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:55.066016 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:55.066543 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 10:50:55.066910 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":354,"threads_started":1}
I20260806 10:50:55.069036 32221 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:50:55.069407 32221 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:55.069512 32221 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe6995a967140c08251a12b812e7f01, State: Initialized, Role: FOLLOWER
I20260806 10:50:55.070058 32221 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:50:55.070295 32221 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:50:55.070458 32221 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:50:55.070660 32221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:55.072904 32221 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:50:55.073464 32221 leader_election.cc:302] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe6995a967140c08251a12b812e7f01; no voters:
I20260806 10:50:55.074158 32221 leader_election.cc:288] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:50:55.074265 32226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:55.074937 32226 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 LEADER]: Becoming Leader. State: Replica: bbe6995a967140c08251a12b812e7f01, State: Running, Role: LEADER
I20260806 10:50:55.075562 32226 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:50:55.076368 32221 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:50:55.078496 32227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbe6995a967140c08251a12b812e7f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } } }
I20260806 10:50:55.078886 32227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:55.078878 32228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe6995a967140c08251a12b812e7f01. Latest consensus state: current_term: 1 leader_uuid: "bbe6995a967140c08251a12b812e7f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } } }
I20260806 10:50:55.081418 32228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: This master's current role is: LEADER
I20260806 10:50:55.082973 32234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:50:55.087989 32234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:50:55.095919 32234 catalog_manager.cc:1347] Generated new cluster ID: c4a9390ded494920bf2b3342ba2aba8e
I20260806 10:50:55.096056 32234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:50:55.117920 32234 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:50:55.119773 32234 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:50:55.129292 32234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Generated new TSK 0
I20260806 10:50:55.130260 32234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:50:55.200012 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40767
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:55.535247 32247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:55.535514 32247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:55.535697 32247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:55.542914 32247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:55.543032 32247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:55.543226 32247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:50:55.552763 32247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:55.553930 32247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:55.555183 32247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:55.568634 32253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:55.569010 32255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:55.569770 32252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:55.569803 32247 server_base.cc:1034] running on GCE node
I20260806 10:50:55.570413 32247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:55.571240 32247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:55.572429 32247 hybrid_clock.cc:648] HybridClock initialized: now 1786013455572383 us; error 40 us; skew 500 ppm
I20260806 10:50:55.572688 32247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:55 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013455, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:55.575716 32247 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:50:55.577210 32247 webserver.cc:460] Webserver started at http://127.16.21.1:44287/ using document root <none> and password file <none>
I20260806 10:50:55.577906 32247 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:55.578070 32247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:55.578382 32247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:55.581673 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "f27f0a706eb44b0289f658ae22fbbaa7"
format_stamp: "Formatted at 2026-08-06 10:50:55 on dist-test-slave-0b58"
server_key: "7f436fadbf65eb908d08274fc46c89c8"
server_key_iv: "71fb5fdc9f8e88e135f964219f47e1aa"
server_key_version: "encryptionkey@0"
I20260806 10:50:55.582418 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "f27f0a706eb44b0289f658ae22fbbaa7"
format_stamp: "Formatted at 2026-08-06 10:50:55 on dist-test-slave-0b58"
server_key: "7f436fadbf65eb908d08274fc46c89c8"
server_key_iv: "71fb5fdc9f8e88e135f964219f47e1aa"
server_key_version: "encryptionkey@0"
I20260806 10:50:55.587236 32247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 10:50:55.590520 32262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:55.591732 32247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:50:55.591885 32247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "f27f0a706eb44b0289f658ae22fbbaa7"
format_stamp: "Formatted at 2026-08-06 10:50:55 on dist-test-slave-0b58"
server_key: "7f436fadbf65eb908d08274fc46c89c8"
server_key_iv: "71fb5fdc9f8e88e135f964219f47e1aa"
server_key_version: "encryptionkey@0"
I20260806 10:50:55.592065 32247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:55.615960 32247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:55.619658 32247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:55.619935 32247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:55.620806 32247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:55.622056 32247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:55.622124 32247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:55.622223 32247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:55.622277 32247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:55.655642 32247 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:34417
I20260806 10:50:55.655717 32374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:34417 every 8 connection(s)
I20260806 10:50:55.657003 32247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:50:55.667395 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 32247
I20260806 10:50:55.667618 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:50:55.668519 16468 external_mini_cluster.cc:1428] Setting key 55694587954fc1baa7220d65ee46a3e2
Aug 06 10:50:55 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013455, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:55.677618 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:55.659566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:36985 (local address 127.16.21.62:40767)
0806 10:50:55.659995 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:55.660000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:55.660985 (+ 985us) server_negotiation.cc:407] Connection header received
0806 10:50:55.662827 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:55.662832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:55.662922 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:55.663044 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:55.664564 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:55.665344 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:55.667036 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:55.667447 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:55.667603 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:55.671727 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:55.671732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:55.671735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:55.671766 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:55.674319 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:55.675053 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:55.675065 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:55.675068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:55.675196 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:55.675613 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:55.675617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:55.675622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:55.675894 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:55.676053 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:55.676963 (+ 910us) server_negotiation.cc:299] Negotiation successful
0806 10:50:55.677191 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":216,"threads_started":1}
I20260806 10:50:55.678504 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:55.660063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:36985)
0806 10:50:55.660838 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:55.660874 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:55.662552 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:55.663276 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:55.663284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:55.663770 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:55.664381 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:55.664400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:55.665504 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:55.665509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:55.666888 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:55.666900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:55.667757 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:55.667767 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:55.668102 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:55.668972 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:55.669003 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:55.671521 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:55.674463 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:55.674471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:55.674485 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:55.674919 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:55.675312 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:55.675317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:55.675321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:55.675496 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:55.676085 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:55.676093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:55.676564 (+ 471us) client_negotiation.cc:769] Sending connection context
0806 10:50:55.676839 (+ 275us) client_negotiation.cc:240] Negotiation successful
0806 10:50:55.677395 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":341,"threads_started":1}
I20260806 10:50:55.680086 32375 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40767
I20260806 10:50:55.680424 32375 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:55.681092 32375 heartbeater.cc:511] Master 127.16.21.62:40767 requested a full tablet report, sending...
I20260806 10:50:55.683122 32186 ts_manager.cc:194] Registered new tserver with Master: f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1:34417)
I20260806 10:50:55.684652 32186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:36985
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:50:55.735361 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40767
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:56.064666 32381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:56.064905 32381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:56.064996 32381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:56.072372 32381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:56.072507 32381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:56.072687 32381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:50:56.082193 32381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:56.083416 32381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:56.084753 32381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:56.097615 32389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:56.097654 32387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:56.097680 32386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:56.098627 32381 server_base.cc:1034] running on GCE node
I20260806 10:50:56.099210 32381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:56.100210 32381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:56.101449 32381 hybrid_clock.cc:648] HybridClock initialized: now 1786013456101395 us; error 52 us; skew 500 ppm
I20260806 10:50:56.101699 32381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:56 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013456, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:56.104683 32381 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:50:56.105993 32381 webserver.cc:460] Webserver started at http://127.16.21.2:35729/ using document root <none> and password file <none>
I20260806 10:50:56.106628 32381 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:56.106786 32381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:56.107081 32381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:56.109850 32381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "a80aae446ee04ae5ad6246e1982d55e3"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "19bfa9f91e0f0d207de804a3179efa94"
server_key_iv: "02f98adb10d864a663c2a417830eda57"
server_key_version: "encryptionkey@0"
I20260806 10:50:56.110553 32381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "a80aae446ee04ae5ad6246e1982d55e3"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "19bfa9f91e0f0d207de804a3179efa94"
server_key_iv: "02f98adb10d864a663c2a417830eda57"
server_key_version: "encryptionkey@0"
I20260806 10:50:56.115383 32381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 10:50:56.118582 32396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:56.119771 32381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:50:56.119942 32381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "a80aae446ee04ae5ad6246e1982d55e3"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "19bfa9f91e0f0d207de804a3179efa94"
server_key_iv: "02f98adb10d864a663c2a417830eda57"
server_key_version: "encryptionkey@0"
I20260806 10:50:56.120132 32381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:56.143656 32381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:56.147465 32381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:56.147755 32381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:56.148749 32381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:56.149956 32381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:56.150058 32381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:56.150211 32381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:56.150293 32381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:56.184135 32381 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:33597
I20260806 10:50:56.184193 32508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:33597 every 8 connection(s)
I20260806 10:50:56.185406 32381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:50:56.191548 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 32381
I20260806 10:50:56.191748 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:50:56.192235 16468 external_mini_cluster.cc:1428] Setting key 339583d33425270a57c22e893db4d0be
Aug 06 10:50:56 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013456, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:56.204978 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.187842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58653 (local address 127.16.21.62:40767)
0806 10:50:56.188324 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.188328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.189224 (+ 896us) server_negotiation.cc:407] Connection header received
0806 10:50:56.190975 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.190979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.191044 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.191172 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:56.192802 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.193718 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.195239 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.195696 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.195846 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.199376 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:56.199382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:56.199388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:56.199426 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:56.201988 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:56.202727 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:56.202733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:56.202736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:56.202874 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:56.203304 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:56.203309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:56.203312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:56.203566 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:56.203729 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.204387 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.204606 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260806 10:50:56.206082 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.188409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:58653)
0806 10:50:56.189072 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.189106 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.190719 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.191346 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.191353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.191962 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:56.192625 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.192645 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.193885 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.193889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.195081 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.195094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.195919 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.195926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.196156 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.196873 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:56.196895 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:56.199157 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:56.202164 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:56.202180 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:56.202193 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:56.202590 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:56.202985 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:56.202991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:56.202994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:56.203183 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:56.203695 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:56.203704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:56.204002 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.204259 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.204752 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
I20260806 10:50:56.207681 32509 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40767
I20260806 10:50:56.208045 32509 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:56.208976 32509 heartbeater.cc:511] Master 127.16.21.62:40767 requested a full tablet report, sending...
I20260806 10:50:56.210788 32186 ts_manager.cc:194] Registered new tserver with Master: a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
I20260806 10:50:56.212123 32186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:58653
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:50:56.269760 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40767
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:50:56.593345 32515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:50:56.593596 32515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:50:56.593794 32515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:50:56.600965 32515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:50:56.601083 32515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:50:56.601271 32515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:50:56.610409 32515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:50:56.611547 32515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:50:56.612828 32515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:50:56.625759 32520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:56.625833 32521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:50:56.625857 32523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:50:56.626797 32515 server_base.cc:1034] running on GCE node
I20260806 10:50:56.627306 32515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:50:56.628122 32515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:50:56.629338 32515 hybrid_clock.cc:648] HybridClock initialized: now 1786013456629294 us; error 59 us; skew 500 ppm
I20260806 10:50:56.629647 32515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:50:56 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013456, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:50:56.632623 32515 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:50:56.633971 32515 webserver.cc:460] Webserver started at http://127.16.21.3:42841/ using document root <none> and password file <none>
I20260806 10:50:56.634619 32515 fs_manager.cc:359] Metadata directory not provided
I20260806 10:50:56.634763 32515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:50:56.635061 32515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:50:56.637926 32515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "a6b5806156844a54a2f02f5a5c5a5e47"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "ec14728b726e59e4c28d9e2f53be4821"
server_key_iv: "a40539f5cb38630754624e4513b2e18d"
server_key_version: "encryptionkey@0"
I20260806 10:50:56.638665 32515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "a6b5806156844a54a2f02f5a5c5a5e47"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "ec14728b726e59e4c28d9e2f53be4821"
server_key_iv: "a40539f5cb38630754624e4513b2e18d"
server_key_version: "encryptionkey@0"
I20260806 10:50:56.643525 32515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 10:50:56.646857 32530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:56.648047 32515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:50:56.648212 32515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "a6b5806156844a54a2f02f5a5c5a5e47"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "ec14728b726e59e4c28d9e2f53be4821"
server_key_iv: "a40539f5cb38630754624e4513b2e18d"
server_key_version: "encryptionkey@0"
I20260806 10:50:56.648461 32515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:50:56.670161 32515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:50:56.674392 32515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:50:56.674676 32515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:50:56.675575 32515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:50:56.676947 32515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:50:56.677047 32515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:56.677214 32515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:50:56.677304 32515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:50:56.687440 32375 heartbeater.cc:503] Master 127.16.21.62:40767 was elected leader, sending a full tablet report...
I20260806 10:50:56.711301 32515 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41419
I20260806 10:50:56.711385 32642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41419 every 8 connection(s)
I20260806 10:50:56.712570 32515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:50:56.715086 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 32515
I20260806 10:50:56.715339 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:50:56.715891 16468 external_mini_cluster.cc:1428] Setting key c63e58a1584473cee8a7b4057994620b
Aug 06 10:50:56 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013456, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:50:56.732722 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.715076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:40059 (local address 127.16.21.62:40767)
0806 10:50:56.715654 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.715660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.716621 (+ 961us) server_negotiation.cc:407] Connection header received
0806 10:50:56.718916 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.718920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.718982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.719097 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:56.720606 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.721406 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.722934 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.723393 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.723535 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.727007 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:56.727011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:56.727015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:56.727047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:56.729539 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:56.730464 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:56.730470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:56.730474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:56.730636 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:56.731149 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:56.731153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:56.731156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:56.731526 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:56.731649 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.732299 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.732468 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":307,"threads_started":1}
I20260806 10:50:56.733568 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.715577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:40059)
0806 10:50:56.716451 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.716494 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.718619 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.719331 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.719339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.719805 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:50:56.720391 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.720433 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.721580 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.721584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.722788 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.722801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.723620 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.723630 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.723874 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.724621 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:50:56.724643 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:50:56.726820 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:50:56.729717 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:56.729723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:56.729733 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:56.730329 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:56.730749 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:50:56.730753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:50:56.730762 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:50:56.731015 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:50:56.731661 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:50:56.731667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:50:56.731943 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.732153 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.732547 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260806 10:50:56.734887 32643 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40767
I20260806 10:50:56.735167 32643 heartbeater.cc:464] Registering TS with master...
I20260806 10:50:56.735819 32643 heartbeater.cc:511] Master 127.16.21.62:40767 requested a full tablet report, sending...
I20260806 10:50:56.737180 32186 ts_manager.cc:194] Registered new tserver with Master: a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419)
I20260806 10:50:56.737964 32186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:40059
I20260806 10:50:56.743376 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:50:56.760409 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.747763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.16.21.62:40767)
0806 10:50:56.747926 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.747933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.748392 (+ 459us) server_negotiation.cc:407] Connection header received
0806 10:50:56.748594 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.748599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.748670 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.748770 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:56.750226 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.750930 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.752562 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.753010 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.753139 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.755470 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:56.755475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:56.755481 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:56.755515 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:56.757804 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:56.758493 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:56.758500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:56.758503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:56.758623 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:56.759039 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:56.759044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:56.759047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:56.759343 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:56.759580 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.759903 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.760104 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:50:56.763929 32186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:50:56.771073 32186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:50:56.801050 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.782535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.16.21.3:41419)
0806 10:50:56.789476 (+ 6941us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.789488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.790688 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 10:50:56.790808 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.790813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.791040 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.791222 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:56.792376 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.794322 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.798005 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.799451 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.799656 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.799889 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.800537 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.800720 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6717,"thread_start_us":6519,"threads_started":1}
I20260806 10:50:56.801138 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.783148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:33597 (local address 127.0.0.1:41408)
0806 10:50:56.790986 (+ 7838us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.791003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.791210 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.792297 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.792303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.792339 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:56.792876 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.792890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.795556 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.795564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.798331 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.798344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.800320 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.800328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.800514 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.800552 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.800655 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.800795 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7700,"thread_start_us":6759,"threads_started":1}
I20260806 10:50:56.801447 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.783457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.16.21.2:33597)
0806 10:50:56.789732 (+ 6275us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.789740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.791090 (+ 1350us) server_negotiation.cc:407] Connection header received
0806 10:50:56.791442 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.791453 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.791724 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.791917 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:56.793138 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.795333 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.798495 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.800079 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.800202 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.800434 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.800935 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.801137 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6111,"thread_start_us":5954,"threads_started":1}
I20260806 10:50:56.801934 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.782283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41419 (local address 127.0.0.1:57726)
0806 10:50:56.790283 (+ 8000us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.790321 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.790498 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.791544 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.791553 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.791582 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:56.792097 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.792114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.794476 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.794481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.797765 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.797786 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.800001 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.800008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.800202 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.800281 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.800411 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.800594 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7756,"mutex_wait_us":1,"thread_start_us":8413,"threads_started":1}
I20260806 10:50:56.805703 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.791061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:38838)
0806 10:50:56.797584 (+ 6523us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.797600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.797858 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.799053 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.799058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.799082 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:56.799698 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.799712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.801826 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.801831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.803505 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.803518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.804929 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.804938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.805133 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.805171 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.805238 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.805367 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6411,"thread_start_us":6617,"threads_started":1}
I20260806 10:50:56.805365 32444 tablet_service.cc:1467] Processing CreateTablet for tablet 808dda7b8d404df8b640c434d0d89168 (DEFAULT_TABLE table=test-table [id=77f4f92335334b3eabdc854cdaa6745e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:56.805457 32578 tablet_service.cc:1467] Processing CreateTablet for tablet 808dda7b8d404df8b640c434d0d89168 (DEFAULT_TABLE table=test-table [id=77f4f92335334b3eabdc854cdaa6745e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:56.806666 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.791479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.16.21.1:34417)
0806 10:50:56.798408 (+ 6929us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.798418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.798447 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:50:56.798534 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.798539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.798722 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.799884 (+ 1162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:56.800080 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.801674 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.803665 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.804701 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.806087 (+ 1386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.806281 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.806334 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.806443 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6749,"thread_start_us":6611,"threads_started":1}
I20260806 10:50:56.806842 32578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808dda7b8d404df8b640c434d0d89168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:56.806844 32444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808dda7b8d404df8b640c434d0d89168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:56.809309 32310 tablet_service.cc:1467] Processing CreateTablet for tablet 808dda7b8d404df8b640c434d0d89168 (DEFAULT_TABLE table=test-table [id=77f4f92335334b3eabdc854cdaa6745e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:50:56.810420 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808dda7b8d404df8b640c434d0d89168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:50:56.822118 32662 tablet_bootstrap.cc:492] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Bootstrap starting.
I20260806 10:50:56.825369 32663 tablet_bootstrap.cc:492] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Bootstrap starting.
I20260806 10:50:56.825714 32662 tablet_bootstrap.cc:654] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:56.826099 32664 tablet_bootstrap.cc:492] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Bootstrap starting.
I20260806 10:50:56.827132 32662 log.cc:824] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:56.828716 32663 tablet_bootstrap.cc:654] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:56.828981 32664 tablet_bootstrap.cc:654] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Neither blocks nor log segments found. Creating new log.
I20260806 10:50:56.829939 32664 log.cc:824] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:56.829972 32663 log.cc:824] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Log is configured to *not* fsync() on all Append() calls
I20260806 10:50:56.830340 32662 tablet_bootstrap.cc:492] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: No bootstrap required, opened a new log
I20260806 10:50:56.830721 32662 ts_tablet_manager.cc:1397] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 10:50:56.832960 32664 tablet_bootstrap.cc:492] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: No bootstrap required, opened a new log
I20260806 10:50:56.832958 32663 tablet_bootstrap.cc:492] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: No bootstrap required, opened a new log
I20260806 10:50:56.833210 32663 ts_tablet_manager.cc:1397] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260806 10:50:56.833210 32664 ts_tablet_manager.cc:1397] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260806 10:50:56.838150 32662 raft_consensus.cc:348] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.838500 32662 raft_consensus.cc:374] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:56.838622 32662 raft_consensus.cc:729] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27f0a706eb44b0289f658ae22fbbaa7, State: Initialized, Role: FOLLOWER
I20260806 10:50:56.839195 32662 consensus_queue.cc:260] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.840888 32662 ts_tablet_manager.cc:1428] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 10:50:56.843729 32663 raft_consensus.cc:348] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.843735 32664 raft_consensus.cc:348] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.844135 32664 raft_consensus.cc:374] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:56.844167 32663 raft_consensus.cc:374] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:50:56.844275 32664 raft_consensus.cc:729] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b5806156844a54a2f02f5a5c5a5e47, State: Initialized, Role: FOLLOWER
I20260806 10:50:56.844313 32663 raft_consensus.cc:729] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80aae446ee04ae5ad6246e1982d55e3, State: Initialized, Role: FOLLOWER
I20260806 10:50:56.844990 32664 consensus_queue.cc:260] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.845022 32663 consensus_queue.cc:260] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.846546 32643 heartbeater.cc:503] Master 127.16.21.62:40767 was elected leader, sending a full tablet report...
I20260806 10:50:56.846596 32509 heartbeater.cc:503] Master 127.16.21.62:40767 was elected leader, sending a full tablet report...
I20260806 10:50:56.846923 32663 ts_tablet_manager.cc:1428] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260806 10:50:56.847162 32664 ts_tablet_manager.cc:1428] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260806 10:50:56.888491 32668 raft_consensus.cc:482] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:50:56.888715 32668 raft_consensus.cc:504] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.890219 32668 leader_election.cc:288] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
I20260806 10:50:56.897351 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.890580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:41301 (local address 127.16.21.3:41419)
0806 10:50:56.890803 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.890808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.891230 (+ 422us) server_negotiation.cc:407] Connection header received
0806 10:50:56.891523 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.891528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.891615 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.891710 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:56.892463 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.893477 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.895499 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.896366 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.896461 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.896584 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.896938 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.897086 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:50:56.897382 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.890403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41419 (local address 127.16.21.1:41301)
0806 10:50:56.891095 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.891126 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.891371 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.891837 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.891842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.891867 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:56.892307 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.892319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.893646 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.893649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.895313 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.895325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.896589 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.896595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.896764 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.896794 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.896869 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.896952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":25,"thread_start_us":331,"threads_started":1}
I20260806 10:50:56.897469 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.891148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:35705 (local address 127.16.21.2:33597)
0806 10:50:56.891364 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.891371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.891635 (+ 264us) server_negotiation.cc:407] Connection header received
0806 10:50:56.891846 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.891852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.891927 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.892020 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:50:56.892693 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.893434 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.895481 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.896406 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.896583 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.896672 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.897081 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.897209 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 10:50:56.897557 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.890666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:33597 (local address 127.16.21.1:35705)
0806 10:50:56.891507 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 10:50:56.891525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:50:56.891645 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:50:56.892062 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:50:56.892065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:50:56.892086 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:50:56.892532 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.892545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.893569 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.893576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.895302 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:50:56.895315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.896745 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.896750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:50:56.896901 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.896932 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:50:56.897029 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 10:50:56.897145 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":84,"thread_start_us":503,"threads_started":1}
I20260806 10:50:56.898682 32464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808dda7b8d404df8b640c434d0d89168" candidate_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80aae446ee04ae5ad6246e1982d55e3" is_pre_election: true
I20260806 10:50:56.898684 32598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808dda7b8d404df8b640c434d0d89168" candidate_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" is_pre_election: true
I20260806 10:50:56.899246 32598 raft_consensus.cc:2454] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27f0a706eb44b0289f658ae22fbbaa7 in term 0.
I20260806 10:50:56.899246 32464 raft_consensus.cc:2454] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27f0a706eb44b0289f658ae22fbbaa7 in term 0.
I20260806 10:50:56.899991 32265 leader_election.cc:302] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80aae446ee04ae5ad6246e1982d55e3, f27f0a706eb44b0289f658ae22fbbaa7; no voters:
I20260806 10:50:56.900470 32668 raft_consensus.cc:2781] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:50:56.900622 32668 raft_consensus.cc:482] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:50:56.900740 32668 raft_consensus.cc:3037] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:56.902808 32668 raft_consensus.cc:504] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:50:56.903498 32668 leader_election.cc:288] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 1 election: Requested vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
I20260806 10:50:56.903883 32598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808dda7b8d404df8b640c434d0d89168" candidate_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6b5806156844a54a2f02f5a5c5a5e47"
I20260806 10:50:56.904048 32464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808dda7b8d404df8b640c434d0d89168" candidate_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80aae446ee04ae5ad6246e1982d55e3"
I20260806 10:50:56.904095 32598 raft_consensus.cc:3037] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:56.904249 32464 raft_consensus.cc:3037] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:50:56.907397 32598 raft_consensus.cc:2454] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f27f0a706eb44b0289f658ae22fbbaa7 in term 1.
I20260806 10:50:56.907433 32464 raft_consensus.cc:2454] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f27f0a706eb44b0289f658ae22fbbaa7 in term 1.
I20260806 10:50:56.907791 32265 leader_election.cc:302] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80aae446ee04ae5ad6246e1982d55e3, f27f0a706eb44b0289f658ae22fbbaa7; no voters:
I20260806 10:50:56.908089 32668 raft_consensus.cc:2781] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:50:56.908558 32668 raft_consensus.cc:686] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 LEADER]: Becoming Leader. State: Replica: f27f0a706eb44b0289f658ae22fbbaa7, State: Running, Role: LEADER
I20260806 10:50:56.909024 32668 consensus_queue.cc:237] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
W20260806 10:50:56.909369 32376 tablet.cc:2369] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:56.913575 32185 catalog_manager.cc:5495] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1). New cstate: current_term: 1 leader_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } health_report { overall_health: UNKNOWN } } }
W20260806 10:50:56.937378 32510 tablet.cc:2369] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:56.946007 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.940616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.16.21.1:34417)
0806 10:50:56.940825 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.940830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:56.940850 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:50:56.940942 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:56.940947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:56.941028 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:56.941191 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:50:56.941578 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.942308 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.943834 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:56.944320 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:56.944463 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:56.944973 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:50:56.945427 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:50:56.945584 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:56.945689 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 10:50:56.945767 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:50:56.953691 32598 raft_consensus.cc:1261] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Refusing update from remote peer f27f0a706eb44b0289f658ae22fbbaa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:56.953699 32464 raft_consensus.cc:1261] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Refusing update from remote peer f27f0a706eb44b0289f658ae22fbbaa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:50:56.954597 32673 consensus_queue.cc:1035] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:50:56.954980 32668 consensus_queue.cc:1035] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 10:50:56.964066 32644 tablet.cc:2369] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:50:56.969223 32677 mvcc.cc:205] Tried to move back new op lower bound from 7315511119672782848 to 7315511119503142912. Current Snapshot: MvccSnapshot[applied={T|T < 7315511119672782848}]
I20260806 10:50:56.982234 32186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:56.982539 32186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:50:56.986404 32186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Sending DeleteTablet for 3 replicas of tablet 808dda7b8d404df8b640c434d0d89168
I20260806 10:50:56.987489 32578 tablet_service.cc:1514] Processing DeleteTablet for tablet 808dda7b8d404df8b640c434d0d89168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:56 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:57726
I20260806 10:50:56.987609 32310 tablet_service.cc:1514] Processing DeleteTablet for tablet 808dda7b8d404df8b640c434d0d89168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:56 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:38838
I20260806 10:50:56.987823 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 32247
I20260806 10:50:56.988808 32444 tablet_service.cc:1514] Processing DeleteTablet for tablet 808dda7b8d404df8b640c434d0d89168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:50:56 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41408
I20260806 10:50:57.004787 32689 tablet_replica.cc:330] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: stopping tablet replica
I20260806 10:50:57.005201 32689 raft_consensus.cc:2229] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:57.005738 32689 raft_consensus.cc:2258] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:57.008450 32689 ts_tablet_manager.cc:1905] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:57.015714 32686 tablet_replica.cc:330] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: stopping tablet replica
I20260806 10:50:57.016175 32686 raft_consensus.cc:2229] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:50:57.016783 32686 raft_consensus.cc:2258] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:57.017804 32689 ts_tablet_manager.cc:1918] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:57.017848 32687 tablet_replica.cc:330] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: stopping tablet replica
I20260806 10:50:57.017968 32689 log.cc:1196] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/808dda7b8d404df8b640c434d0d89168
I20260806 10:50:57.018366 32687 raft_consensus.cc:2229] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:50:57.018442 32689 ts_tablet_manager.cc:1939] T 808dda7b8d404df8b640c434d0d89168 P a80aae446ee04ae5ad6246e1982d55e3: Deleting consensus metadata
I20260806 10:50:57.019093 32686 ts_tablet_manager.cc:1905] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:57.019107 32687 raft_consensus.cc:2258] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:50:57.020120 32172 catalog_manager.cc:4841] TS a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): tablet 808dda7b8d404df8b640c434d0d89168 (table test-table [id=77f4f92335334b3eabdc854cdaa6745e]) successfully deleted
I20260806 10:50:57.021062 32687 ts_tablet_manager.cc:1905] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:50:57.028033 32686 ts_tablet_manager.cc:1918] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:57.028198 32686 log.cc:1196] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/808dda7b8d404df8b640c434d0d89168
I20260806 10:50:57.028630 32687 ts_tablet_manager.cc:1918] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:50:57.028743 32686 ts_tablet_manager.cc:1939] T 808dda7b8d404df8b640c434d0d89168 P a6b5806156844a54a2f02f5a5c5a5e47: Deleting consensus metadata
I20260806 10:50:57.028822 32687 log.cc:1196] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/808dda7b8d404df8b640c434d0d89168
I20260806 10:50:57.029397 32687 ts_tablet_manager.cc:1939] T 808dda7b8d404df8b640c434d0d89168 P f27f0a706eb44b0289f658ae22fbbaa7: Deleting consensus metadata
I20260806 10:50:57.030480 32170 catalog_manager.cc:4841] TS a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): tablet 808dda7b8d404df8b640c434d0d89168 (table test-table [id=77f4f92335334b3eabdc854cdaa6745e]) successfully deleted
Aug 06 10:50:57 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:50:57.030977 32170 catalog_manager.cc:4841] TS f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1:34417): tablet 808dda7b8d404df8b640c434d0d89168 (table test-table [id=77f4f92335334b3eabdc854cdaa6745e]) successfully deleted
I20260806 10:50:57.038976 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:56.988914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.16.21.1:34417)
0806 10:50:56.989087 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:56.989093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:57.014864 (+ 25771us) server_negotiation.cc:407] Connection header received
0806 10:50:57.015105 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:57.015111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:57.015190 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:57.015289 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:57.021412 (+ 6123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:57.022611 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:57.024904 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:57.025664 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:57.025780 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:57.031970 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:57.032002 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:57.032020 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:57.032069 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:57.035890 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:57.036713 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:57.036721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:57.036726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:57.036900 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:57.037342 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:57.037348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:57.037351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:57.037787 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:57.038057 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:57.038359 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 10:50:57.038612 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:50:57.039688 32370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:57.139079 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 32247
I20260806 10:50:57.161042 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 32381
Aug 06 10:50:57 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:50:57.177217 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:57.161836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.16.21.2:33597)
0806 10:50:57.162022 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:57.162029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:57.162129 (+ 100us) server_negotiation.cc:407] Connection header received
0806 10:50:57.162350 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:57.162356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:57.162429 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:57.162561 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:57.163908 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:57.164803 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:57.166617 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:57.167032 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:57.167171 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:57.171660 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:57.171673 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:57.171687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:57.171723 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:57.174414 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:57.175081 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:57.175088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:57.175090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:57.175216 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:57.175594 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:57.175600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:57.175603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:57.175969 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:57.176335 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:57.176636 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 10:50:57.176880 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:50:57.177956 32504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:57.299710 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 32381
I20260806 10:50:57.319541 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 32515
Aug 06 10:50:57 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:50:57.334786 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:50:57.320380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.16.21.3:41419)
0806 10:50:57.320561 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 10:50:57.320565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:50:57.320719 (+ 154us) server_negotiation.cc:407] Connection header received
0806 10:50:57.320800 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:50:57.320804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:50:57.320878 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:50:57.320976 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:50:57.322247 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:57.323014 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:57.324442 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:50:57.324850 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:50:57.324946 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:50:57.329280 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:50:57.329296 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:50:57.329312 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:50:57.329354 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:50:57.332340 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:57.332936 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:57.332943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:57.332946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:57.333086 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:50:57.333432 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:50:57.333435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:50:57.333437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:50:57.333767 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:50:57.333977 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 10:50:57.334258 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 10:50:57.334428 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:50:57.335434 32638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:57.425820 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 32515
I20260806 10:50:57.445986 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 32154
I20260806 10:50:57.447255 32216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:50:57.556138 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 32154
I20260806 10:51:17.583671 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40767
--webserver_interface=127.16.21.62
--webserver_port=39039
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:51:17.929973 32694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:17.930234 32694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:17.930330 32694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:17.937389 32694 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 10:51:17.937479 32694 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 10:51:17.937551 32694 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 10:51:17.937651 32694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:51:17.937741 32694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:17.937861 32694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:17.937940 32694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:51:17.938023 32694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:51:17.947789 32694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:17.948993 32694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:17.950271 32694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:17.976172 32703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:17.976150 32701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:17.976244 32700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:17.977042 32694 server_base.cc:1034] running on GCE node
I20260806 10:51:17.977931 32694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:17.979511 32694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:17.980785 32694 hybrid_clock.cc:648] HybridClock initialized: now 1786013477980709 us; error 62 us; skew 500 ppm
I20260806 10:51:17.981182 32694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:17 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:17.984902 32694 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:51:17.986522 32694 webserver.cc:460] Webserver started at http://127.16.21.62:39039/ using document root <none> and password file <none>
I20260806 10:51:17.987323 32694 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:17.987478 32694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:17.993422 32694 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:51:17.996495 32710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:17.997826 32694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:51:17.998029 32694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "bbe6995a967140c08251a12b812e7f01"
format_stamp: "Formatted at 2026-08-06 10:50:54 on dist-test-slave-0b58"
server_key: "932a111ff47f0d8165a1e3785d3075ef"
server_key_iv: "b70b3e170bf5b40f9608d65d5cc40925"
server_key_version: "encryptionkey@0"
I20260806 10:51:17.999099 32694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:18.013862 32694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:18.017948 32694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:18.018256 32694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:18.038518 32694 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40767
I20260806 10:51:18.038586 32762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40767 every 8 connection(s)
I20260806 10:51:18.040141 32694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:51:18.046154 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 32694
I20260806 10:51:18.047258 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:34417
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:40767
--webserver_port=44287
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:51:18.047912 32763 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:51:18.052995 32763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap starting.
I20260806 10:51:18.076396 32763 log.cc:824] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Log is configured to *not* fsync() on all Append() calls
I20260806 10:51:18.088748 32763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:51:18.089313 32763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap complete.
I20260806 10:51:18.098574 32763 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:18.099400 32763 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe6995a967140c08251a12b812e7f01, State: Initialized, Role: FOLLOWER
I20260806 10:51:18.100004 32763 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:18.100246 32763 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:51:18.100430 32763 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:51:18.100600 32763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:51:18.104586 32763 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:18.105177 32763 leader_election.cc:302] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe6995a967140c08251a12b812e7f01; no voters:
I20260806 10:51:18.105989 32763 leader_election.cc:288] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:51:18.106148 32767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:51:18.108456 32763 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:51:18.113003 32767 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 LEADER]: Becoming Leader. State: Replica: bbe6995a967140c08251a12b812e7f01, State: Running, Role: LEADER
I20260806 10:51:18.113612 32767 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:18.117350 301 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe6995a967140c08251a12b812e7f01. Latest consensus state: current_term: 2 leader_uuid: "bbe6995a967140c08251a12b812e7f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } } }
I20260806 10:51:18.117710 301 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: This master's current role is: LEADER
I20260806 10:51:18.118243 300 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bbe6995a967140c08251a12b812e7f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } } }
I20260806 10:51:18.118597 300 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: This master's current role is: LEADER
I20260806 10:51:18.128470 314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:51:18.136528 314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:51:18.137902 314 catalog_manager.cc:1259] Loaded cluster ID: c4a9390ded494920bf2b3342ba2aba8e
I20260806 10:51:18.137995 314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:51:18.142060 314 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:51:18.143556 314 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Loaded TSK: 0
I20260806 10:51:18.173862 314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Generated new TSK 1
I20260806 10:51:18.174378 314 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:51:18.412040 32765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:18.412446 32765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:18.412643 32765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:18.419989 32765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:18.420135 32765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:18.420358 32765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:51:18.430058 32765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=44287
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:18.431735 32765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:18.433136 32765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:18.447993 323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:18.448539 322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:18.448915 32765 server_base.cc:1034] running on GCE node
W20260806 10:51:18.448930 325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:18.449826 32765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:18.450855 32765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:18.452085 32765 hybrid_clock.cc:648] HybridClock initialized: now 1786013478452021 us; error 52 us; skew 500 ppm
I20260806 10:51:18.452453 32765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:18 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:18.459852 32765 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:51:18.461508 32765 webserver.cc:460] Webserver started at http://127.16.21.1:44287/ using document root <none> and password file <none>
I20260806 10:51:18.462186 32765 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:18.462333 32765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:18.467708 32765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 10:51:18.470489 332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:18.471846 32765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:51:18.472023 32765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "f27f0a706eb44b0289f658ae22fbbaa7"
format_stamp: "Formatted at 2026-08-06 10:50:55 on dist-test-slave-0b58"
server_key: "7f436fadbf65eb908d08274fc46c89c8"
server_key_iv: "71fb5fdc9f8e88e135f964219f47e1aa"
server_key_version: "encryptionkey@0"
I20260806 10:51:18.473001 32765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:18.497783 32765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:18.501878 32765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:18.502206 32765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:18.503258 32765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:51:18.504736 32765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:51:18.504864 32765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:18.504973 32765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:51:18.505095 32765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:18.538537 32765 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:34417
I20260806 10:51:18.538621 445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:34417 every 8 connection(s)
I20260806 10:51:18.539901 32765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:51:18.542796 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 32765
I20260806 10:51:18.543635 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:33597
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:40767
--webserver_port=35729
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:51:18 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:51:18.567066 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:18.542928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:36649)
0806 10:51:18.543699 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:18.543767 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:18.545727 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:18.547124 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:18.547134 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:18.547730 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:51:18.548438 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:18.548494 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:18.550394 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:18.550402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:18.552590 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:18.552609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:18.553632 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:18.553641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:18.553960 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:18.554710 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:51:18.554745 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:51:18.557761 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:51:18.562737 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:18.562745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:18.562758 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:18.563207 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:18.563743 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:18.563747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:18.563749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:18.563937 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:18.565002 (+ 1065us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:51:18.565009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:51:18.565334 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 10:51:18.565579 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 10:51:18.565998 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":345,"threads_started":1}
I20260806 10:51:18.567617 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:18.542614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:36649 (local address 127.16.21.62:40767)
0806 10:51:18.543202 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:18.543213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:18.544009 (+ 796us) server_negotiation.cc:407] Connection header received
0806 10:51:18.546292 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:18.546306 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:18.546607 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:18.546977 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:18.548741 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:18.550203 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:18.552745 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:18.553395 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:18.553541 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:18.558139 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:18.558152 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:18.558175 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:18.558238 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:18.562568 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:18.563382 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:18.563394 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:18.563404 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:18.563626 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:18.564071 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:18.564079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:18.564085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:18.564837 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:18.565039 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:18.565853 (+ 814us) server_negotiation.cc:299] Negotiation successful
0806 10:51:18.566401 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
I20260806 10:51:18.570477 446 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40767
I20260806 10:51:18.570817 446 heartbeater.cc:464] Registering TS with master...
I20260806 10:51:18.571537 446 heartbeater.cc:511] Master 127.16.21.62:40767 requested a full tablet report, sending...
I20260806 10:51:18.573721 32728 ts_manager.cc:194] Registered new tserver with Master: f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1:34417)
I20260806 10:51:18.575371 32728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:36649
W20260806 10:51:18.887683 450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:18.887948 450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:18.888063 450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:18.895121 450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:18.895229 450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:18.895433 450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:51:18.905395 450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:33597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=35729
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:18.906641 450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:18.907932 450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:18.921375 455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:18.921432 456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:18.921624 458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:18.922441 450 server_base.cc:1034] running on GCE node
I20260806 10:51:18.922971 450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:18.923815 450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:18.925045 450 hybrid_clock.cc:648] HybridClock initialized: now 1786013478924992 us; error 46 us; skew 500 ppm
I20260806 10:51:18.925347 450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:18 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:18.928411 450 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:51:18.929853 450 webserver.cc:460] Webserver started at http://127.16.21.2:35729/ using document root <none> and password file <none>
I20260806 10:51:18.930447 450 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:18.930557 450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:18.935457 450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:51:18.938022 465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:18.939293 450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:51:18.939424 450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "a80aae446ee04ae5ad6246e1982d55e3"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "19bfa9f91e0f0d207de804a3179efa94"
server_key_iv: "02f98adb10d864a663c2a417830eda57"
server_key_version: "encryptionkey@0"
I20260806 10:51:18.940373 450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:18.961936 450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:18.969152 450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:18.969476 450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:18.970748 450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:51:18.972618 450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:51:18.972781 450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:18.972905 450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:51:18.972991 450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:19.008808 450 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:33597
I20260806 10:51:19.008836 578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:33597 every 8 connection(s)
I20260806 10:51:19.010890 450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:51:19.011206 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 450
I20260806 10:51:19.012043 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:41419
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:40767
--webserver_port=42841
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:51:19 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:51:19.048501 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.018196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:38197 (local address 127.16.21.62:40767)
0806 10:51:19.018453 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.018462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.025168 (+ 6706us) server_negotiation.cc:407] Connection header received
0806 10:51:19.028582 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.028591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.028695 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.028831 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:19.030344 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.031016 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.033592 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.034416 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.034529 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.039870 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:19.039879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:19.039887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:19.039936 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:19.044528 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.045713 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.045719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.045722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.045956 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.046514 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.046519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.046523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.046886 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:19.047011 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.047880 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.048119 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 10:51:19.050853 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.017646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:38197)
0806 10:51:19.024948 (+ 7302us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.025011 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.028194 (+ 3183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.029144 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.029152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.029643 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:51:19.030164 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.030180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.031174 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.031182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.033425 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.033442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.034878 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.034887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.035229 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.036056 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:51:19.036095 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:51:19.039527 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:51:19.044667 (+ 5140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:19.044781 (+ 114us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:19.044796 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:19.045557 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:19.046111 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:19.046116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:19.046119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:19.046386 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:19.047096 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:51:19.047104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:51:19.047505 (+ 401us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.049020 (+ 1515us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.049541 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6953,"thread_start_us":6689,"threads_started":1}
I20260806 10:51:19.052620 579 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40767
I20260806 10:51:19.052955 579 heartbeater.cc:464] Registering TS with master...
I20260806 10:51:19.053625 579 heartbeater.cc:511] Master 127.16.21.62:40767 requested a full tablet report, sending...
I20260806 10:51:19.055004 32728 ts_manager.cc:194] Registered new tserver with Master: a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
I20260806 10:51:19.056156 32728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:38197
W20260806 10:51:19.391466 582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:19.391707 582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:19.391806 582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:19.398828 582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:19.398921 582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:19.399031 582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:51:19.408339 582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=42841
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:19.409431 582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:19.410642 582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:19.423693 590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:19.423754 587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:19.423720 588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:19.424762 582 server_base.cc:1034] running on GCE node
I20260806 10:51:19.425343 582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:19.426286 582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:19.427508 582 hybrid_clock.cc:648] HybridClock initialized: now 1786013479427469 us; error 30 us; skew 500 ppm
I20260806 10:51:19.427804 582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:19 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:19.430753 582 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:51:19.432196 582 webserver.cc:460] Webserver started at http://127.16.21.3:42841/ using document root <none> and password file <none>
I20260806 10:51:19.432921 582 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:19.433066 582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:19.438102 582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 10:51:19.440713 597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:19.442016 582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:51:19.442162 582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "a6b5806156844a54a2f02f5a5c5a5e47"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "ec14728b726e59e4c28d9e2f53be4821"
server_key_iv: "a40539f5cb38630754624e4513b2e18d"
server_key_version: "encryptionkey@0"
I20260806 10:51:19.443125 582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:19.458105 582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:19.462126 582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:19.462333 582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:19.463186 582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:51:19.464443 582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:51:19.464581 582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:19.464684 582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:51:19.464764 582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:19.497095 582 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41419
I20260806 10:51:19.497179 710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41419 every 8 connection(s)
I20260806 10:51:19.498500 582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:51:19.500051 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 582
Aug 06 10:51:19 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013479, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:51:19.520306 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.501173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.16.21.62:40767)
0806 10:51:19.506173 (+ 5000us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.506180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.506199 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:51:19.506264 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.506269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.506349 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.506461 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:19.508089 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.509241 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.510605 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.511261 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.511394 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.514375 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:19.514380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:19.514384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:19.514413 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:19.517293 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.518172 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.518176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.518178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.518332 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.518797 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.518804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.518808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.519140 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:19.519413 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.519754 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.519938 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4847,"thread_start_us":4716,"threads_started":1}
I20260806 10:51:19.523789 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.501054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:58177 (local address 127.16.21.62:40767)
0806 10:51:19.501271 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.501279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.505208 (+ 3929us) server_negotiation.cc:407] Connection header received
0806 10:51:19.507564 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.507570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.507646 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.507746 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:19.509624 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.510394 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.512575 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.513152 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.513285 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.516976 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:19.516982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:19.516985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:19.517027 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:19.520490 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.521402 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.521408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.521413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.521560 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.522048 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.522052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.522054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.522302 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:19.522406 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.523290 (+ 884us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.523539 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 10:51:19.524808 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.501559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:58177)
0806 10:51:19.504994 (+ 3435us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.505061 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.507286 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.507994 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.508001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.508703 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:51:19.509478 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.509496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.510543 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.510548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.512433 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.512446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.513400 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.513408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.513636 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.514425 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:51:19.514446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:51:19.516754 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:51:19.520649 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:19.520657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:19.520673 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:19.521260 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:19.521695 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:19.521700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:19.521704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:19.521926 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:19.522434 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:51:19.522441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:51:19.522874 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.523185 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.523618 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3056,"thread_start_us":2798,"threads_started":1}
I20260806 10:51:19.526237 711 heartbeater.cc:347] Connected to a master server at 127.16.21.62:40767
I20260806 10:51:19.526536 711 heartbeater.cc:464] Registering TS with master...
I20260806 10:51:19.527241 711 heartbeater.cc:511] Master 127.16.21.62:40767 requested a full tablet report, sending...
I20260806 10:51:19.528692 32728 ts_manager.cc:194] Registered new tserver with Master: a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419)
I20260806 10:51:19.529497 32728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:58177
I20260806 10:51:19.535126 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:51:19.541824 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.535978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.16.21.62:40767)
0806 10:51:19.536102 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.536106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.536688 (+ 582us) server_negotiation.cc:407] Connection header received
0806 10:51:19.536920 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.536924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.537037 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.537181 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:51:19.537660 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.538398 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.539945 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.540405 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.540543 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.541039 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:51:19.541223 (+ 184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 10:51:19.541565 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50340: token expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 10:51:19.541965 448 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50340: token expired
W20260806 10:51:19.542598 32647 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.21.62:40767)
I20260806 10:51:19.555178 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.543136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.16.21.62:40767)
0806 10:51:19.543268 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.543272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.543288 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:51:19.543398 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.543401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.543461 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.543532 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:19.545196 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.545905 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.547224 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.547707 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.547803 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.550363 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:19.550369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:19.550372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:19.550400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:19.553021 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.553650 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.553654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.553656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.553760 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.554084 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.554087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.554089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.554356 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:19.554525 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.554809 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.554964 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 10:51:19.558317 32728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:51:19.565868 32728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:51:19.578621 446 heartbeater.cc:503] Master 127.16.21.62:40767 was elected leader, sending a full tablet report...
I20260806 10:51:19.599448 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.578138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:33597 (local address 127.0.0.1:36424)
0806 10:51:19.592542 (+ 14404us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.592558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.592687 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.593394 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.593398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.593422 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:51:19.593853 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.593862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.595217 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.595222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.597113 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.597126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.598430 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.598436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.598646 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.598699 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.598826 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.599008 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14287,"mutex_wait_us":26,"thread_start_us":14155,"threads_started":1}
I20260806 10:51:19.599629 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.578375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.16.21.2:33597)
0806 10:51:19.592208 (+ 13833us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.592219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.592833 (+ 614us) server_negotiation.cc:407] Connection header received
0806 10:51:19.592944 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.592952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.593163 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.593407 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:51:19.594026 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.595077 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.597264 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.598196 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.598370 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.598593 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.598879 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.599254 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13678,"thread_start_us":13524,"threads_started":1}
I20260806 10:51:19.600669 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.577630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55386)
0806 10:51:19.588630 (+ 11000us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.588667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.588842 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.589694 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.589705 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.589744 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:51:19.590271 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.590289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.593360 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.593368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.596960 (+ 3592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.596977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.599522 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.599531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.599769 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.599823 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.599899 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.600074 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10767,"mutex_wait_us":1,"thread_start_us":10563,"threads_started":1}
I20260806 10:51:19.601919 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.577833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.16.21.1:34417)
0806 10:51:19.588101 (+ 10268us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.588112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.589038 (+ 926us) server_negotiation.cc:407] Connection header received
0806 10:51:19.589155 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.589165 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.589432 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.590528 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:51:19.590808 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.592928 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.597215 (+ 4287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.599051 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.600936 (+ 1885us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.601177 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.601277 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.601473 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10096,"thread_start_us":9941,"threads_started":1}
I20260806 10:51:19.604183 514 tablet_service.cc:1467] Processing CreateTablet for tablet de7eaf5502404c7f91a6186f4af84c15 (DEFAULT_TABLE table=test-table [id=2af363c29bf640449d11e48c6315e118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:51:19.605690 514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7eaf5502404c7f91a6186f4af84c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:51:19.605981 381 tablet_service.cc:1467] Processing CreateTablet for tablet de7eaf5502404c7f91a6186f4af84c15 (DEFAULT_TABLE table=test-table [id=2af363c29bf640449d11e48c6315e118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:51:19.607362 381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7eaf5502404c7f91a6186f4af84c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:51:19.608950 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.589409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.16.21.3:41419)
0806 10:51:19.602986 (+ 13577us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.602992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.603016 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:51:19.603088 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.603092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.603242 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.603446 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:51:19.603878 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.605318 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.606923 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.607883 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.608024 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.608238 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.608477 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.608620 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13396,"thread_start_us":13227,"threads_started":1}
I20260806 10:51:19.609562 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.595292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41419 (local address 127.0.0.1:42170)
0806 10:51:19.599998 (+ 4706us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.600015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.600107 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.603440 (+ 3333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.603444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.603469 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:51:19.603654 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.603689 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.605454 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.605460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.606772 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.606784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.608037 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.608043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.608232 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.608319 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.608425 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.608556 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4624,"thread_start_us":7052,"threads_started":1}
I20260806 10:51:19.612612 646 tablet_service.cc:1467] Processing CreateTablet for tablet de7eaf5502404c7f91a6186f4af84c15 (DEFAULT_TABLE table=test-table [id=2af363c29bf640449d11e48c6315e118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:51:19.613783 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7eaf5502404c7f91a6186f4af84c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:51:19.623289 727 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Bootstrap starting.
I20260806 10:51:19.626408 729 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Bootstrap starting.
I20260806 10:51:19.627285 728 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Bootstrap starting.
I20260806 10:51:19.627771 727 tablet_bootstrap.cc:654] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Neither blocks nor log segments found. Creating new log.
I20260806 10:51:19.629196 727 log.cc:824] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Log is configured to *not* fsync() on all Append() calls
I20260806 10:51:19.629436 729 tablet_bootstrap.cc:654] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Neither blocks nor log segments found. Creating new log.
I20260806 10:51:19.630618 728 tablet_bootstrap.cc:654] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Neither blocks nor log segments found. Creating new log.
I20260806 10:51:19.630667 729 log.cc:824] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Log is configured to *not* fsync() on all Append() calls
I20260806 10:51:19.631886 728 log.cc:824] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:51:19.633055 727 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: No bootstrap required, opened a new log
I20260806 10:51:19.633363 727 ts_tablet_manager.cc:1397] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 10:51:19.633527 729 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: No bootstrap required, opened a new log
I20260806 10:51:19.633766 729 ts_tablet_manager.cc:1397] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:51:19.634878 728 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: No bootstrap required, opened a new log
I20260806 10:51:19.635128 728 ts_tablet_manager.cc:1397] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:51:19.640447 729 raft_consensus.cc:348] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.640790 729 raft_consensus.cc:374] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:51:19.640890 729 raft_consensus.cc:729] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b5806156844a54a2f02f5a5c5a5e47, State: Initialized, Role: FOLLOWER
I20260806 10:51:19.641667 729 consensus_queue.cc:260] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.643282 711 heartbeater.cc:503] Master 127.16.21.62:40767 was elected leader, sending a full tablet report...
I20260806 10:51:19.643919 729 ts_tablet_manager.cc:1428] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260806 10:51:19.644665 727 raft_consensus.cc:348] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.645072 727 raft_consensus.cc:374] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:51:19.645179 727 raft_consensus.cc:729] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80aae446ee04ae5ad6246e1982d55e3, State: Initialized, Role: FOLLOWER
I20260806 10:51:19.645679 728 raft_consensus.cc:348] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.645771 727 consensus_queue.cc:260] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.646091 728 raft_consensus.cc:374] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:51:19.646232 728 raft_consensus.cc:729] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27f0a706eb44b0289f658ae22fbbaa7, State: Initialized, Role: FOLLOWER
I20260806 10:51:19.646999 728 consensus_queue.cc:260] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.647464 579 heartbeater.cc:503] Master 127.16.21.62:40767 was elected leader, sending a full tablet report...
I20260806 10:51:19.647636 727 ts_tablet_manager.cc:1428] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260806 10:51:19.648687 728 ts_tablet_manager.cc:1428] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260806 10:51:19.685770 733 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:19.686066 733 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.687732 733 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1:34417), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
I20260806 10:51:19.694552 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.688052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.16.21.3:52119)
0806 10:51:19.688310 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.688330 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.688455 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.688995 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.689001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.689038 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:51:19.689245 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.689264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.690637 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.690640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.692605 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.692622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.693945 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.693949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.694082 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.694105 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.694181 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.694278 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 10:51:19.694880 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.688345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:52119 (local address 127.16.21.1:34417)
0806 10:51:19.688541 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.688548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.688566 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:51:19.688642 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.688649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.688729 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.688831 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:51:19.689403 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.690501 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.692765 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.693805 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.693935 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.694032 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.694235 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.694375 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:51:19.695266 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.688175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:33597 (local address 127.16.21.3:35705)
0806 10:51:19.689041 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.689059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:19.689190 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:19.689584 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:19.689588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:19.689612 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:51:19.689926 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.689934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.691110 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.691116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.693021 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:19.693031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.694607 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.694614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:19.694781 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.694804 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:51:19.694874 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 10:51:19.694986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":501,"threads_started":1}
I20260806 10:51:19.695335 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.688355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:35705 (local address 127.16.21.2:33597)
0806 10:51:19.688612 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.688621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.689159 (+ 538us) server_negotiation.cc:407] Connection header received
0806 10:51:19.689362 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.689368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.689444 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.689532 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:51:19.690105 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.690981 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.693174 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.694385 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.694522 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.694607 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.694943 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.695080 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:51:19.695865 401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7eaf5502404c7f91a6186f4af84c15" candidate_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" is_pre_election: true
I20260806 10:51:19.696457 401 raft_consensus.cc:2454] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b5806156844a54a2f02f5a5c5a5e47 in term 0.
I20260806 10:51:19.696653 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7eaf5502404c7f91a6186f4af84c15" candidate_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80aae446ee04ae5ad6246e1982d55e3" is_pre_election: true
I20260806 10:51:19.697124 534 raft_consensus.cc:2454] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b5806156844a54a2f02f5a5c5a5e47 in term 0.
I20260806 10:51:19.697175 598 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b5806156844a54a2f02f5a5c5a5e47, f27f0a706eb44b0289f658ae22fbbaa7; no voters:
I20260806 10:51:19.697618 733 raft_consensus.cc:2781] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:51:19.697773 733 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:51:19.697901 733 raft_consensus.cc:3037] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:51:19.700129 733 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.700774 733 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [CANDIDATE]: Term 1 election: Requested vote from peers f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1:34417), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
I20260806 10:51:19.701212 401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7eaf5502404c7f91a6186f4af84c15" candidate_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27f0a706eb44b0289f658ae22fbbaa7"
I20260806 10:51:19.701236 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7eaf5502404c7f91a6186f4af84c15" candidate_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80aae446ee04ae5ad6246e1982d55e3"
I20260806 10:51:19.701426 401 raft_consensus.cc:3037] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:51:19.701422 534 raft_consensus.cc:3037] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:51:19.703684 534 raft_consensus.cc:2454] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b5806156844a54a2f02f5a5c5a5e47 in term 1.
I20260806 10:51:19.704033 401 raft_consensus.cc:2454] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b5806156844a54a2f02f5a5c5a5e47 in term 1.
I20260806 10:51:19.704062 600 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3; no voters:
I20260806 10:51:19.704355 733 raft_consensus.cc:2781] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:51:19.704809 733 raft_consensus.cc:686] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 LEADER]: Becoming Leader. State: Replica: a6b5806156844a54a2f02f5a5c5a5e47, State: Running, Role: LEADER
I20260806 10:51:19.705252 733 consensus_queue.cc:237] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:19.709914 32727 catalog_manager.cc:5495] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3). New cstate: current_term: 1 leader_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } health_report { overall_health: UNKNOWN } } }
I20260806 10:51:19.725350 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.719969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.16.21.3:41419)
0806 10:51:19.720158 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.720164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.720258 (+ 94us) server_negotiation.cc:407] Connection header received
0806 10:51:19.720527 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.720531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.720601 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.720744 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:51:19.721122 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.721885 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.723241 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.723644 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.723775 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.724341 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:51:19.724788 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:51:19.724938 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.725022 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.725102 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:51:19.732824 401 raft_consensus.cc:1261] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Refusing update from remote peer a6b5806156844a54a2f02f5a5c5a5e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:51:19.732858 534 raft_consensus.cc:1261] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Refusing update from remote peer a6b5806156844a54a2f02f5a5c5a5e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:51:19.733736 738 consensus_queue.cc:1035] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:51:19.734071 733 consensus_queue.cc:1035] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:51:19.757678 32727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:51:19.757987 32727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:51:19.761250 32727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Sending DeleteTablet for 3 replicas of tablet de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:19.762380 646 tablet_service.cc:1514] Processing DeleteTablet for tablet de7eaf5502404c7f91a6186f4af84c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:51:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:42170
I20260806 10:51:19.762458 381 tablet_service.cc:1514] Processing DeleteTablet for tablet de7eaf5502404c7f91a6186f4af84c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:51:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:55386
I20260806 10:51:19.762609 514 tablet_service.cc:1514] Processing DeleteTablet for tablet de7eaf5502404c7f91a6186f4af84c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:51:19 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:36424
I20260806 10:51:19.763657 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 32765
I20260806 10:51:19.777078 753 tablet_replica.cc:330] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: stopping tablet replica
I20260806 10:51:19.778178 753 raft_consensus.cc:2229] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:51:19.779075 753 raft_consensus.cc:2258] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:51:19.781929 753 ts_tablet_manager.cc:1905] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:51:19.786545 751 tablet_replica.cc:330] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: stopping tablet replica
I20260806 10:51:19.786933 751 raft_consensus.cc:2229] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:51:19.787407 751 raft_consensus.cc:2258] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:51:19.788587 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.764565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.16.21.1:34417)
0806 10:51:19.764728 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.764734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.765050 (+ 316us) server_negotiation.cc:407] Connection header received
0806 10:51:19.765138 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.765146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.765257 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.765385 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:19.767887 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.769504 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.772180 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.773133 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.773277 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.778267 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:19.778287 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:19.778308 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:19.778359 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:19.783541 (+ 5182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.784752 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.784779 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.784785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.784972 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.785549 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.785557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.785563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.786139 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:19.786497 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.787726 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.788046 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:51:19.789290 441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:19.791378 753 ts_tablet_manager.cc:1918] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:51:19.791582 753 log.cc:1196] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:19.792229 753 ts_tablet_manager.cc:1939] T de7eaf5502404c7f91a6186f4af84c15 P a80aae446ee04ae5ad6246e1982d55e3: Deleting consensus metadata
I20260806 10:51:19.794178 32714 catalog_manager.cc:4841] TS a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): tablet de7eaf5502404c7f91a6186f4af84c15 (table test-table [id=2af363c29bf640449d11e48c6315e118]) successfully deleted
I20260806 10:51:19.795167 752 tablet_replica.cc:330] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: stopping tablet replica
I20260806 10:51:19.795635 752 raft_consensus.cc:2229] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:51:19.796413 752 raft_consensus.cc:2258] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:51:19.798552 752 ts_tablet_manager.cc:1905] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:51:19.806316 752 ts_tablet_manager.cc:1918] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:51:19.806499 752 log.cc:1196] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:19.807097 752 ts_tablet_manager.cc:1939] T de7eaf5502404c7f91a6186f4af84c15 P a6b5806156844a54a2f02f5a5c5a5e47: Deleting consensus metadata
I20260806 10:51:19.808871 32712 catalog_manager.cc:4841] TS a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): tablet de7eaf5502404c7f91a6186f4af84c15 (table test-table [id=2af363c29bf640449d11e48c6315e118]) successfully deleted
I20260806 10:51:19.889175 751 ts_tablet_manager.cc:1905] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:51:19.890693 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 32765
I20260806 10:51:19.912976 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 450
W20260806 10:51:19.913036 32712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.21.1:34417 (error 108)
I20260806 10:51:19.915169 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.914676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55418)
0806 10:51:19.914868 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.915000 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 10:51:19.915503 32712 catalog_manager.cc:4584] TS f27f0a706eb44b0289f658ae22fbbaa7 (127.16.21.1:34417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de7eaf5502404c7f91a6186f4af84c15: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
I20260806 10:51:19.929140 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.914353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.16.21.2:33597)
0806 10:51:19.914520 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:19.914524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:19.914540 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:51:19.914593 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:19.914596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:19.914666 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:19.914786 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:19.916694 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.917550 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.919596 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:19.919994 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:19.920081 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:19.923453 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:19.923466 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:19.923480 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:19.923510 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:19.926427 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.927277 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.927285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.927287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.927414 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:19.927783 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:19.927786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:19.927788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:19.928124 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:19.928398 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:19.928714 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 10:51:19.928868 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:51:19.929780 574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:19.969069 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.968560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55430)
0806 10:51:19.968787 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.968892 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 10:51:19.970221 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:19.969796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55436)
0806 10:51:19.969958 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:19.970065 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 10:51:20.023025 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 450
I20260806 10:51:20.031086 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:20.030533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55448)
0806 10:51:20.030786 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:20.030895 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 10:51:20.032343 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:20.031893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55458)
0806 10:51:20.032075 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:20.032170 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 10:51:20.045964 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 582
I20260806 10:51:20.064881 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:20.046865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.16.21.3:41419)
0806 10:51:20.047057 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:20.047065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:20.047088 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:51:20.047298 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:20.047304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:20.047422 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:20.047542 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:20.049205 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:20.050418 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:20.052162 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:20.052838 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:20.052978 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:20.056244 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:20.056291 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:20.056311 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:20.056361 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:20.060540 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:20.061495 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:20.061504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:20.061508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:20.061714 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:20.062170 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:20.062175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:20.062181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:20.062719 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:20.063024 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:20.064193 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0806 10:51:20.064447 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 10:51:20.065737 706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:20.113129 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:20.112626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55474)
0806 10:51:20.112845 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:20.112952 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 10:51:20.114347 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:20.113816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55486)
0806 10:51:20.114089 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:20.114174 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260806 10:51:20.189407 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 582
I20260806 10:51:20.216884 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 32694
I20260806 10:51:20.218044 32758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:20.330858 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:20.330065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:34417 (local address 127.0.0.1:55490)
0806 10:51:20.330434 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:20.330579 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 10:51:20.331015 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 32694
I20260806 10:51:40.357089 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40767
--webserver_interface=127.16.21.62
--webserver_port=39039
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:51:40.694939 759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:40.695219 759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:40.695403 759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:40.702481 759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 10:51:40.702562 759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 10:51:40.702662 759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 10:51:40.702790 759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:51:40.702890 759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:40.703015 759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:40.703087 759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:51:40.703181 759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:51:40.712821 759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:40767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:40.714043 759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:40.715327 759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:40.727214 767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:40.727347 764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:40.727311 765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:40.727599 759 server_base.cc:1034] running on GCE node
I20260806 10:51:40.728668 759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:40.729979 759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:40.731228 759 hybrid_clock.cc:648] HybridClock initialized: now 1786013500731160 us; error 42 us; skew 500 ppm
I20260806 10:51:40.731549 759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:40 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013500, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:40.734710 759 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:51:40.736081 759 webserver.cc:460] Webserver started at http://127.16.21.62:39039/ using document root <none> and password file <none>
I20260806 10:51:40.736794 759 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:40.736928 759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:40.742031 759 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 10:51:40.744704 774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:40.746100 759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:51:40.746299 759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "bbe6995a967140c08251a12b812e7f01"
format_stamp: "Formatted at 2026-08-06 10:50:54 on dist-test-slave-0b58"
server_key: "932a111ff47f0d8165a1e3785d3075ef"
server_key_iv: "b70b3e170bf5b40f9608d65d5cc40925"
server_key_version: "encryptionkey@0"
I20260806 10:51:40.747227 759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:40.779135 759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:40.783121 759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:40.783413 759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:40.802597 759 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:40767
I20260806 10:51:40.802629 826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:40767 every 8 connection(s)
I20260806 10:51:40.804055 759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:51:40.809228 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 759
I20260806 10:51:40.810053 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:34417
--local_ip_for_outbound_sockets=127.16.21.1
--tserver_master_addrs=127.16.21.62:40767
--webserver_port=44287
--webserver_interface=127.16.21.1
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:51:40.811152 827 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:51:40.814463 827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap starting.
I20260806 10:51:40.835870 827 log.cc:824] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Log is configured to *not* fsync() on all Append() calls
I20260806 10:51:40.852645 827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:51:40.853178 827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Bootstrap complete.
I20260806 10:51:40.862529 827 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:40.863420 827 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe6995a967140c08251a12b812e7f01, State: Initialized, Role: FOLLOWER
I20260806 10:51:40.864102 827 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:40.864380 827 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:51:40.864518 827 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:51:40.864712 827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 2 FOLLOWER]: Advancing to term 3
I20260806 10:51:40.868556 827 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:40.869061 827 leader_election.cc:302] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe6995a967140c08251a12b812e7f01; no voters:
I20260806 10:51:40.869746 827 leader_election.cc:288] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 10:51:40.869899 831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 10:51:40.871279 831 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [term 3 LEADER]: Becoming Leader. State: Replica: bbe6995a967140c08251a12b812e7f01, State: Running, Role: LEADER
I20260806 10:51:40.871917 831 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } }
I20260806 10:51:40.872275 827 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:51:40.874600 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe6995a967140c08251a12b812e7f01. Latest consensus state: current_term: 3 leader_uuid: "bbe6995a967140c08251a12b812e7f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } } }
I20260806 10:51:40.875033 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: This master's current role is: LEADER
I20260806 10:51:40.874822 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bbe6995a967140c08251a12b812e7f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe6995a967140c08251a12b812e7f01" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 40767 } } }
I20260806 10:51:40.875245 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01 [sys.catalog]: This master's current role is: LEADER
I20260806 10:51:40.876232 838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:51:40.888116 838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:51:40.891855 838 catalog_manager.cc:1259] Loaded cluster ID: c4a9390ded494920bf2b3342ba2aba8e
I20260806 10:51:40.892023 838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:51:40.896472 838 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:51:40.898280 838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Loaded TSK: 0
I20260806 10:51:40.898502 838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbe6995a967140c08251a12b812e7f01: Loaded TSK: 1
F20260806 10:51:40.900044 838 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9cd31099ad google::LogMessage::Fail() at ??:0
@ 0x7f9cd310c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9cd31094de google::LogMessage::Flush() at ??:0
@ 0x7f9cd310aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9cd6e6d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9cd6e5c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9ce8b381c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9ce8b37783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9ce8bc32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9ce8bc30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9ce8c0d5e9 std::function<>::operator()() at ??:0
@ 0x7f9ce8b3bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9ce8b39c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9ce8bc1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9ce8bc1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9cd5d55345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9cd5d5891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9cd5d58799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9cd5d33eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9cd900d6db start_thread at ??:0
@ 0x7f9cd1fa361f clone at ??:0
W20260806 10:51:41.238605 829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:41.238869 829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:41.238965 829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:41.246399 829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:41.246496 829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:41.246615 829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:51:41.256042 829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=44287
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:41.257270 829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:41.258463 829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:41.271454 878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:41.271574 876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:41.271545 875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:41.272701 829 server_base.cc:1034] running on GCE node
I20260806 10:51:41.273312 829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:41.274191 829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:41.275424 829 hybrid_clock.cc:648] HybridClock initialized: now 1786013501275387 us; error 48 us; skew 500 ppm
I20260806 10:51:41.275729 829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:41 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013501, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:41.278802 829 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:51:41.280206 829 webserver.cc:460] Webserver started at http://127.16.21.1:44287/ using document root <none> and password file <none>
I20260806 10:51:41.280896 829 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:41.281045 829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:41.286154 829 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 10:51:41.288968 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:41.290177 829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:51:41.290357 829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "f27f0a706eb44b0289f658ae22fbbaa7"
format_stamp: "Formatted at 2026-08-06 10:50:55 on dist-test-slave-0b58"
server_key: "7f436fadbf65eb908d08274fc46c89c8"
server_key_iv: "71fb5fdc9f8e88e135f964219f47e1aa"
server_key_version: "encryptionkey@0"
I20260806 10:51:41.291304 829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:41.306890 829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:41.310583 829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:41.310873 829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:41.311801 829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:51:41.313572 892 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 10:51:41.318926 829 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 10:51:41.319072 829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260806 10:51:41.319200 829 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 10:51:41.322228 829 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 10:51:41.322335 829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260806 10:51:41.322683 892 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Bootstrap starting.
I20260806 10:51:41.330426 892 log.cc:824] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:51:41.345021 892 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:51:41.346027 892 tablet_bootstrap.cc:492] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Bootstrap complete.
I20260806 10:51:41.346541 892 ts_tablet_manager.cc:1397] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20260806 10:51:41.356194 892 raft_consensus.cc:348] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:41.357968 892 raft_consensus.cc:729] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27f0a706eb44b0289f658ae22fbbaa7, State: Initialized, Role: FOLLOWER
I20260806 10:51:41.358598 892 consensus_queue.cc:260] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:41.360085 892 ts_tablet_manager.cc:1428] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260806 10:51:41.366477 829 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:34417
I20260806 10:51:41.366550 1000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:34417 every 8 connection(s)
I20260806 10:51:41.367653 829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:51:41.371850 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:41.370106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:51065)
0806 10:51:41.370834 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:41.371040 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":303,"threads_started":1}
W20260806 10:51:41.372534 1001 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:40767 (0 consecutive failures): Network error: Failed to ping master at 127.16.21.62:40767: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
I20260806 10:51:41.373327 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:41.372987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:41225)
0806 10:51:41.373147 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:41.373208 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 10:51:41.373435 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 829
I20260806 10:51:41.374094 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:33597
--local_ip_for_outbound_sockets=127.16.21.2
--tserver_master_addrs=127.16.21.62:40767
--webserver_port=35729
--webserver_interface=127.16.21.2
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:51:41.377071 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:41.376710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:38741)
0806 10:51:41.376873 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:41.376954 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 10:51:41.377360 1001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:51:41.713027 1004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:41.713277 1004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:41.713469 1004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:41.720666 1004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:41.720796 1004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:41.720993 1004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:51:41.730232 1004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:33597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=35729
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:41.731473 1004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:41.732790 1004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:41.745944 1010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:41.746104 1009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:41.746064 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:41.747071 1004 server_base.cc:1034] running on GCE node
I20260806 10:51:41.747588 1004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:41.748485 1004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:41.749686 1004 hybrid_clock.cc:648] HybridClock initialized: now 1786013501749655 us; error 30 us; skew 500 ppm
I20260806 10:51:41.749938 1004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:41 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013501, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:41.757983 1004 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:51:41.759277 1004 webserver.cc:460] Webserver started at http://127.16.21.2:35729/ using document root <none> and password file <none>
I20260806 10:51:41.759867 1004 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:41.759963 1004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:41.765010 1004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:51:41.767570 1020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:41.768900 1004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:51:41.769030 1004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "a80aae446ee04ae5ad6246e1982d55e3"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "19bfa9f91e0f0d207de804a3179efa94"
server_key_iv: "02f98adb10d864a663c2a417830eda57"
server_key_version: "encryptionkey@0"
I20260806 10:51:41.769934 1004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:41.796474 1004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:41.800064 1004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:41.800333 1004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:41.801189 1004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:51:41.802372 1004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:51:41.802454 1004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:41.802525 1004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:51:41.802579 1004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:41.836086 1004 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:33597
I20260806 10:51:41.836122 1133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:33597 every 8 connection(s)
I20260806 10:51:41.837527 1004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:51:41.841965 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:41.840204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:52181)
0806 10:51:41.840972 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:41.841188 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":283,"threads_started":1}
W20260806 10:51:41.842612 1134 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:40767 (0 consecutive failures): Network error: Failed to ping master at 127.16.21.62:40767: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
I20260806 10:51:41.843325 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:41.843041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:39147)
0806 10:51:41.843161 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:41.843222 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 10:51:41.844074 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:41.843786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:49585)
0806 10:51:41.843904 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:41.843973 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:51:41.844355 1134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:51:41.845572 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1004
I20260806 10:51:41.846227 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:41419
--local_ip_for_outbound_sockets=127.16.21.3
--tserver_master_addrs=127.16.21.62:40767
--webserver_port=42841
--webserver_interface=127.16.21.3
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:51:42.181434 1137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:42.181694 1137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:42.181872 1137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:42.189011 1137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:42.189173 1137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:42.189375 1137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:51:42.199038 1137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=42841
--tserver_master_addrs=127.16.21.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:42.200191 1137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:42.201469 1137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:42.214052 1143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:42.214021 1142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:42.214114 1145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:42.215044 1137 server_base.cc:1034] running on GCE node
I20260806 10:51:42.215673 1137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:42.216642 1137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:42.217837 1137 hybrid_clock.cc:648] HybridClock initialized: now 1786013502217804 us; error 27 us; skew 500 ppm
I20260806 10:51:42.218133 1137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:42 dist-test-slave-0b58 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013502, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:42.221354 1137 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:51:42.222883 1137 webserver.cc:460] Webserver started at http://127.16.21.3:42841/ using document root <none> and password file <none>
I20260806 10:51:42.223856 1137 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:42.224007 1137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:42.230434 1137 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:51:42.233421 1153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:42.234632 1137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:51:42.234793 1137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "a6b5806156844a54a2f02f5a5c5a5e47"
format_stamp: "Formatted at 2026-08-06 10:50:56 on dist-test-slave-0b58"
server_key: "ec14728b726e59e4c28d9e2f53be4821"
server_key_iv: "a40539f5cb38630754624e4513b2e18d"
server_key_version: "encryptionkey@0"
I20260806 10:51:42.235755 1137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:42.253561 1137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:42.257038 1137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:42.257315 1137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:42.258219 1137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:51:42.259461 1137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:51:42.259567 1137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:42.259698 1137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:51:42.259797 1137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:42.294154 1137 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41419
I20260806 10:51:42.294296 1266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41419 every 8 connection(s)
I20260806 10:51:42.295650 1137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:51:42.298115 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1137
I20260806 10:51:42.300686 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.298586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:42217)
0806 10:51:42.299409 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.299660 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":364,"threads_started":1}
W20260806 10:51:42.301338 1267 heartbeater.cc:644] Failed to heartbeat to 127.16.21.62:40767 (0 consecutive failures): Network error: Failed to ping master at 127.16.21.62:40767: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
I20260806 10:51:42.302076 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.301786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:44845)
0806 10:51:42.301916 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.301979 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 10:51:42.302891 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.302608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:33357)
0806 10:51:42.302721 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.302795 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:51:42.303162 1267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:51:42.378544 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.377822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:37857)
0806 10:51:42.378215 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.378298 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":191,"threads_started":1}
I20260806 10:51:42.645849 1272 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:42.646121 1272 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:42.647923 1272 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
Aug 06 10:51:42 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013501, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
Aug 06 10:51:42 dist-test-slave-0b58 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013501, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:51:42.670737 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.648328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:33597 (local address 127.16.21.1:39451)
0806 10:51:42.649932 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.649955 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:42.650071 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:42.653148 (+ 3077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:42.653152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:42.653687 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:51:42.654409 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:42.654422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.655972 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.655976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:42.657393 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:42.657405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.658378 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.658385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:42.658915 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:42.659679 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:51:42.659709 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:51:42.662847 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:51:42.667169 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:42.667175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:42.667179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:42.667586 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:42.668133 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:42.668137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:42.668140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:42.668357 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:42.669449 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:51:42.669454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:51:42.669681 (+ 227us) client_negotiation.cc:769] Sending connection context
0806 10:51:42.670014 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 10:51:42.670411 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"mutex_wait_us":108,"spinlock_wait_cycles":4736,"thread_start_us":1226,"threads_started":1}
I20260806 10:51:42.671123 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.648033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41419 (local address 127.16.21.1:41023)
0806 10:51:42.648392 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.648449 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 10:51:42.649054 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:51:42.652926 (+ 3872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:51:42.652936 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:51:42.653678 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:51:42.654398 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:42.654419 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.655970 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.655974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:42.657399 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:51:42.657412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.658551 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.658558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:51:42.658764 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:42.659679 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:51:42.659733 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:51:42.662963 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:51:42.666938 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:42.666947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:42.666961 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:42.667502 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:42.667993 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:51:42.667999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:51:42.668003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:51:42.668327 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:51:42.669318 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:51:42.669327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:51:42.669683 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 10:51:42.670015 (+ 332us) client_negotiation.cc:240] Negotiation successful
0806 10:51:42.670602 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":317,"spinlock_wait_cycles":34944}
I20260806 10:51:42.671679 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.648517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:41023 (local address 127.16.21.3:41419)
0806 10:51:42.650426 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:42.650441 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:42.650488 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:51:42.652009 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:42.652018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:42.652453 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:42.652717 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:42.654602 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.655787 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.657537 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.658309 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.659258 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:42.663212 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:42.663219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:42.663231 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:42.663270 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:42.666776 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:42.667658 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:42.667667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:42.667675 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:42.667862 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:42.668479 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:42.668485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:42.668488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:42.669145 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:42.669359 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:42.670475 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0806 10:51:42.671268 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1698,"thread_start_us":1450,"threads_started":1}
I20260806 10:51:42.672049 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.648556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:39451 (local address 127.16.21.2:33597)
0806 10:51:42.650539 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:42.650550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:42.650579 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:51:42.652025 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:42.652033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:42.652454 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:42.652791 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:42.654611 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.655771 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.657532 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:42.658186 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:42.658384 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:42.663173 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:42.663181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:42.663195 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:42.663232 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:42.667026 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:42.667786 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:42.667797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:42.667807 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:42.668010 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:42.668479 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:42.668485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:42.668488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:42.669145 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:42.669743 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:42.671232 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0806 10:51:42.671635 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1839,"thread_start_us":1729,"threads_started":1}
W20260806 10:51:42.675302 888 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
W20260806 10:51:42.675552 886 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:42.675736 886 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f27f0a706eb44b0289f658ae22fbbaa7; no voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3
I20260806 10:51:42.676170 1272 raft_consensus.cc:2726] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:51:42.845876 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:42.844912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:37087)
0806 10:51:42.845456 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:42.845612 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":265,"threads_started":1}
I20260806 10:51:43.304374 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:43.303612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:56989)
0806 10:51:43.304004 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:43.304103 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":194,"threads_started":1}
I20260806 10:51:43.379947 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:43.379318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:54705)
0806 10:51:43.379726 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:43.379802 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":200,"threads_started":1}
I20260806 10:51:43.847313 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:43.846664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:46239)
0806 10:51:43.847088 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:43.847166 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":234,"threads_started":1}
I20260806 10:51:44.305795 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:44.305134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:43127)
0806 10:51:44.305552 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:44.305625 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260806 10:51:44.381510 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:44.380796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:43603)
0806 10:51:44.381231 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:44.381306 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260806 10:51:44.449903 1282 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:44.450078 1282 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:44.450618 1282 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
W20260806 10:51:44.451678 888 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
W20260806 10:51:44.451910 886 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:44.452047 886 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f27f0a706eb44b0289f658ae22fbbaa7; no voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3
I20260806 10:51:44.452378 1282 raft_consensus.cc:2726] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:51:44.848758 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:44.848097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:33239)
0806 10:51:44.848530 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:44.848605 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":231,"threads_started":1}
I20260806 10:51:45.307345 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:45.306671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:40255)
0806 10:51:45.307098 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:45.307183 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260806 10:51:45.383044 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:45.382389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:46985)
0806 10:51:45.382809 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:45.382885 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":229,"threads_started":1}
I20260806 10:51:45.850320 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:45.849612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:42241)
0806 10:51:45.850051 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:45.850132 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260806 10:51:46.005059 1287 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:46.005261 1287 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:46.005960 1287 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
W20260806 10:51:46.006942 886 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
W20260806 10:51:46.007103 888 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:46.007272 888 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f27f0a706eb44b0289f658ae22fbbaa7; no voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3
I20260806 10:51:46.007520 1287 raft_consensus.cc:2726] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:51:46.308904 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:46.308160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:57187)
0806 10:51:46.308630 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:46.308715 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":221,"threads_started":1}
I20260806 10:51:46.385200 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:46.384183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:45883)
0806 10:51:46.384871 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:46.384980 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":265,"threads_started":1}
I20260806 10:51:46.851852 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:46.851203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:58925)
0806 10:51:46.851630 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:46.851708 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260806 10:51:47.310483 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:47.309713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:41497)
0806 10:51:47.310236 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:47.310319 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":301,"threads_started":1}
I20260806 10:51:47.386741 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:47.386052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:50637)
0806 10:51:47.386492 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:47.386576 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260806 10:51:47.853435 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:47.852703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:33665)
0806 10:51:47.853182 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:47.853264 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260806 10:51:48.312034 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:48.311320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:48967)
0806 10:51:48.311768 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:48.311858 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
I20260806 10:51:48.388360 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:48.387587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:38889)
0806 10:51:48.388004 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:48.388083 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260806 10:51:48.855059 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:48.854286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:51001)
0806 10:51:48.854799 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:48.854876 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":242,"threads_started":1}
I20260806 10:51:49.313791 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:49.312981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:37929)
0806 10:51:49.313500 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:49.313576 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":252,"threads_started":1}
I20260806 10:51:49.389832 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:49.389124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:39187)
0806 10:51:49.389561 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:49.389647 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":215,"threads_started":1}
I20260806 10:51:49.856699 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:49.855974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:34591)
0806 10:51:49.856412 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:49.856496 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":213,"threads_started":1}
I20260806 10:51:50.315294 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:50.314621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:52135)
0806 10:51:50.315027 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:50.315106 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":214,"threads_started":1}
I20260806 10:51:50.391333 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:50.390654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:56793)
0806 10:51:50.391088 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:50.391162 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260806 10:51:50.858561 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:50.857742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:33565)
0806 10:51:50.858292 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:50.858387 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260806 10:51:50.914037 1303 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:50.914266 1303 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:50.914928 1303 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
W20260806 10:51:50.915920 888 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
W20260806 10:51:50.916112 886 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:50.916326 886 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f27f0a706eb44b0289f658ae22fbbaa7; no voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3
I20260806 10:51:50.916607 1303 raft_consensus.cc:2726] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:51:51.316921 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:51.316158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:50541)
0806 10:51:51.316684 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:51.316772 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
I20260806 10:51:51.393003 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:51.392223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:48701)
0806 10:51:51.392693 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:51.392806 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":207,"threads_started":1}
I20260806 10:51:51.860946 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:51.859796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:47041)
0806 10:51:51.860491 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:51.860626 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":310,"threads_started":1}
I20260806 10:51:52.323585 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:52.319730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:52215)
0806 10:51:52.322036 (+ 2306us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:52.322416 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1707,"thread_start_us":1265,"threads_started":1}
I20260806 10:51:52.395344 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:52.394072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:60191)
0806 10:51:52.394922 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:52.395072 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":482,"threads_started":1}
I20260806 10:51:52.862721 1309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:52.861983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:34529)
0806 10:51:52.862476 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:52.862564 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":239,"threads_started":1}
I20260806 10:51:52.931443 1310 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:52.931681 1310 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:52.932363 1310 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
W20260806 10:51:52.933331 886 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
W20260806 10:51:52.933521 888 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:52.933688 888 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f27f0a706eb44b0289f658ae22fbbaa7; no voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3
I20260806 10:51:52.933970 1310 raft_consensus.cc:2726] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:51:53.325853 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:53.325172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:58723)
0806 10:51:53.325609 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:53.325682 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":214,"threads_started":1}
I20260806 10:51:53.397132 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:53.396371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:43759)
0806 10:51:53.396842 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:53.396925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":184,"threads_started":1}
I20260806 10:51:53.864408 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:53.863646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:50233)
0806 10:51:53.864099 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:53.864173 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260806 10:51:54.327557 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:54.326864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:41285)
0806 10:51:54.327299 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:54.327375 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260806 10:51:54.398729 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:54.398020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:55267)
0806 10:51:54.398476 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:54.398560 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":259,"threads_started":1}
I20260806 10:51:54.866073 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:54.865377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:53739)
0806 10:51:54.865830 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:54.865915 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260806 10:51:55.329190 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:55.328494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:43013)
0806 10:51:55.328949 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:55.329034 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":224,"threads_started":1}
I20260806 10:51:55.400161 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:55.399532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:45169)
0806 10:51:55.399925 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:55.400003 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":199,"threads_started":1}
I20260806 10:51:55.695554 1319 raft_consensus.cc:482] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:51:55.695850 1319 raft_consensus.cc:504] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27f0a706eb44b0289f658ae22fbbaa7" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 34417 } } peers { permanent_uuid: "a6b5806156844a54a2f02f5a5c5a5e47" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41419 } } peers { permanent_uuid: "a80aae446ee04ae5ad6246e1982d55e3" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 33597 } }
I20260806 10:51:55.696616 1319 leader_election.cc:288] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419), a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597)
W20260806 10:51:55.697594 886 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b5806156844a54a2f02f5a5c5a5e47 (127.16.21.3:41419): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
W20260806 10:51:55.697778 888 leader_election.cc:341] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a80aae446ee04ae5ad6246e1982d55e3 (127.16.21.2:33597): Not found: Tablet not found: de7eaf5502404c7f91a6186f4af84c15
I20260806 10:51:55.697899 888 leader_election.cc:302] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f27f0a706eb44b0289f658ae22fbbaa7; no voters: a6b5806156844a54a2f02f5a5c5a5e47, a80aae446ee04ae5ad6246e1982d55e3
I20260806 10:51:55.698128 1319 raft_consensus.cc:2726] T de7eaf5502404c7f91a6186f4af84c15 P f27f0a706eb44b0289f658ae22fbbaa7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:51:55.867666 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:55.866941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:44601)
0806 10:51:55.867398 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:55.867476 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260806 10:51:56.330667 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:56.329969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:56155)
0806 10:51:56.330400 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:56.330474 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
I20260806 10:51:56.401907 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:56.401232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.1:53267)
0806 10:51:56.401637 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:56.401721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":207,"threads_started":1}
I20260806 10:51:56.869493 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:56.868847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.2:41625)
0806 10:51:56.869250 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:56.869351 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":214,"threads_started":1}
I20260806 10:51:57.309131 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 829
I20260806 10:51:57.328063 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:57.310682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.16.21.1:34417)
0806 10:51:57.311425 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:57.311432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:57.311458 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:51:57.311565 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:57.311572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:57.311792 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:57.311925 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:57.314110 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:57.315120 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:57.317179 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:57.317728 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:57.317838 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:57.321670 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:57.321682 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:57.321696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:57.321732 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:57.325148 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:57.326005 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:57.326012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:57.326015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:57.326146 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:57.326573 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:57.326576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:57.326578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:57.326919 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:57.327145 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:57.327520 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 10:51:57.327708 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":449,"threads_started":1}
I20260806 10:51:57.329473 996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:57.332599 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:57.331660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:40767 (local address 127.16.21.3:33417)
0806 10:51:57.332199 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 10:51:57.332377 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.62:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":262,"threads_started":1}
I20260806 10:51:57.427364 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 829
I20260806 10:51:57.446573 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1004
I20260806 10:51:57.460103 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:57.447378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.16.21.2:33597)
0806 10:51:57.447987 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:57.447992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:57.448007 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:51:57.448062 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:57.448065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:57.448125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:57.448282 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:57.449622 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:57.450409 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:57.451691 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:57.452125 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:57.452330 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:57.454955 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:57.454960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:57.454963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:57.454993 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:57.457655 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:57.458312 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:57.458317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:57.458319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:57.458459 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:57.458787 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:57.458790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:57.458792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:57.459062 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:57.459273 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:57.459504 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 10:51:57.459694 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":362,"threads_started":1}
I20260806 10:51:57.460820 1129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:57.546977 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1004
I20260806 10:51:57.567737 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1137
I20260806 10:51:57.581658 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:57.568598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.16.21.3:41419)
0806 10:51:57.569176 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:57.569181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:57.569197 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:51:57.569251 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:57.569255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:57.569336 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:57.569465 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:57.570863 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:57.571697 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:57.573021 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:57.573455 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:57.573595 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:57.576247 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:57.576293 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:57.576298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:57.576340 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:57.578939 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:57.579604 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:57.579611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:57.579615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:57.579754 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:57.580130 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:57.580136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:57.580139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:57.580473 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:57.580758 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:57.581044 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 10:51:57.581272 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":288,"threads_started":1}
I20260806 10:51:57.582389 1262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:51:57.666685 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1137
2026-08-06T10:51:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64464 ms)
[----------] 30 tests from SecurityITest (295705 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:51:57 dist-test-slave-0b58 krb5kdc[1334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:51:57 dist-test-slave-0b58 krb5kdc[1334](info): set up 2 sockets
Aug 06 10:51:57 dist-test-slave-0b58 krb5kdc[1334](info): commencing operation
krb5kdc: starting...
W20260806 10:51:58.804126 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:51:58 dist-test-slave-0b58 krb5kdc[1334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:51:59.087602 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45285
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:51:59.413738 1347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:51:59.414029 1347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:51:59.414207 1347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:51:59.421308 1347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:51:59.421443 1347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:51:59.421546 1347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:51:59.421633 1347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:51:59.421698 1347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:51:59.431300 1347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:51:59.432483 1347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:51:59.433712 1347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:51:59.445459 1355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:59.445425 1353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:51:59.445922 1352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:51:59.446266 1347 server_base.cc:1034] running on GCE node
I20260806 10:51:59.446962 1347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:51:59.448187 1347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:51:59.449440 1347 hybrid_clock.cc:648] HybridClock initialized: now 1786013519449403 us; error 31 us; skew 500 ppm
I20260806 10:51:59.449767 1347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:51:59 dist-test-slave-0b58 krb5kdc[1334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013519, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:51:59.452927 1347 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:51:59.454218 1347 webserver.cc:460] Webserver started at http://127.16.21.62:39725/ using document root <none> and password file <none>
I20260806 10:51:59.454792 1347 fs_manager.cc:359] Metadata directory not provided
I20260806 10:51:59.454890 1347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:51:59.455128 1347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:51:59.457901 1347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "e91055b6fbe34fa78b431d43b807d2a4"
format_stamp: "Formatted at 2026-08-06 10:51:59 on dist-test-slave-0b58"
server_key: "bcf538fb2e15a997263b400d6c057d96"
server_key_iv: "0b7ec57209bbb6c8dcf9959033c59267"
server_key_version: "encryptionkey@0"
I20260806 10:51:59.458580 1347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "e91055b6fbe34fa78b431d43b807d2a4"
format_stamp: "Formatted at 2026-08-06 10:51:59 on dist-test-slave-0b58"
server_key: "bcf538fb2e15a997263b400d6c057d96"
server_key_iv: "0b7ec57209bbb6c8dcf9959033c59267"
server_key_version: "encryptionkey@0"
I20260806 10:51:59.463371 1347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 10:51:59.466557 1362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:51:59.467742 1347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:51:59.467866 1347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "e91055b6fbe34fa78b431d43b807d2a4"
format_stamp: "Formatted at 2026-08-06 10:51:59 on dist-test-slave-0b58"
server_key: "bcf538fb2e15a997263b400d6c057d96"
server_key_iv: "0b7ec57209bbb6c8dcf9959033c59267"
server_key_version: "encryptionkey@0"
I20260806 10:51:59.468012 1347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:51:59.479643 1347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:51:59.483703 1347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:51:59.483922 1347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:51:59.502839 1347 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:45285
I20260806 10:51:59.502875 1415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:45285 every 8 connection(s)
I20260806 10:51:59.504159 1347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:51:59.507853 1416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:51:59.510579 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1347
I20260806 10:51:59.510804 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:51:59.511637 16468 external_mini_cluster.cc:1428] Setting key 96df12d1043f83bd0c116a27462f57bc
I20260806 10:51:59.519972 1416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4: Bootstrap starting.
I20260806 10:51:59.523391 1416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4: Neither blocks nor log segments found. Creating new log.
I20260806 10:51:59.524672 1416 log.cc:824] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:51:59 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:51:59.527498 1416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4: No bootstrap required, opened a new log
I20260806 10:51:59.534009 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:51:59.513658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.16.21.62:45285)
0806 10:51:59.514376 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 10:51:59.514385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:51:59.514418 (+ 33us) server_negotiation.cc:407] Connection header received
0806 10:51:59.515957 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:51:59.515975 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:51:59.516361 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:51:59.516651 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:51:59.518337 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:59.519615 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:59.521382 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:51:59.521968 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:51:59.522076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:51:59.526932 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:51:59.526939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:51:59.526951 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:51:59.526983 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:51:59.530106 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:59.530801 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:59.530808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:59.530816 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:59.530980 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:51:59.531384 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:51:59.531394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:51:59.531396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:51:59.531884 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:51:59.532250 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 10:51:59.532725 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 10:51:59.533091 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":343,"threads_started":1}
I20260806 10:51:59.535249 1416 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91055b6fbe34fa78b431d43b807d2a4" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45285 } }
I20260806 10:51:59.535622 1416 raft_consensus.cc:374] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:51:59.535737 1416 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91055b6fbe34fa78b431d43b807d2a4, State: Initialized, Role: FOLLOWER
I20260806 10:51:59.536235 1416 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91055b6fbe34fa78b431d43b807d2a4" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45285 } }
I20260806 10:51:59.536545 1416 raft_consensus.cc:388] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:51:59.536684 1416 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:51:59.536856 1416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:51:59.538826 1416 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91055b6fbe34fa78b431d43b807d2a4" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45285 } }
I20260806 10:51:59.539230 1416 leader_election.cc:302] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e91055b6fbe34fa78b431d43b807d2a4; no voters:
I20260806 10:51:59.539750 1416 leader_election.cc:288] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:51:59.539871 1421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:51:59.540518 1421 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [term 1 LEADER]: Becoming Leader. State: Replica: e91055b6fbe34fa78b431d43b807d2a4, State: Running, Role: LEADER
I20260806 10:51:59.541035 1421 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91055b6fbe34fa78b431d43b807d2a4" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45285 } }
I20260806 10:51:59.541611 1416 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:51:59.543658 1423 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91055b6fbe34fa78b431d43b807d2a4. Latest consensus state: current_term: 1 leader_uuid: "e91055b6fbe34fa78b431d43b807d2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91055b6fbe34fa78b431d43b807d2a4" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45285 } } }
I20260806 10:51:59.543989 1423 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [sys.catalog]: This master's current role is: LEADER
I20260806 10:51:59.544399 1422 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e91055b6fbe34fa78b431d43b807d2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91055b6fbe34fa78b431d43b807d2a4" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45285 } } }
I20260806 10:51:59.544704 1422 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4 [sys.catalog]: This master's current role is: LEADER
I20260806 10:51:59.545544 1427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:51:59.550503 1427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:51:59.559538 1427 catalog_manager.cc:1347] Generated new cluster ID: c26b9b92542c4c508acbc80da57d91fc
I20260806 10:51:59.559651 1427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:51:59.605566 1427 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:51:59.607407 1427 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:51:59.615309 1427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4: Generated new TSK 0
I20260806 10:51:59.616034 1427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:51:59.705760 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45285
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:00.038314 1442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:00.038583 1442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:00.038759 1442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:00.046008 1442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:00.046123 1442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:00.046326 1442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:52:00.055338 1442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:00.056481 1442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:00.057618 1442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:00.070256 1447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:00.070317 1448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:00.070374 1450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:00.071172 1442 server_base.cc:1034] running on GCE node
I20260806 10:52:00.071709 1442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:00.072683 1442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:00.073887 1442 hybrid_clock.cc:648] HybridClock initialized: now 1786013520073840 us; error 38 us; skew 500 ppm
I20260806 10:52:00.074173 1442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:00 dist-test-slave-0b58 krb5kdc[1334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013520, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:00.077086 1442 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:52:00.078398 1442 webserver.cc:460] Webserver started at http://127.16.21.1:33479/ using document root <none> and password file <none>
I20260806 10:52:00.079051 1442 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:00.079198 1442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:00.079512 1442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:00.082357 1442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "85e5fb14afd54d068f0db84eb5e054b3"
format_stamp: "Formatted at 2026-08-06 10:52:00 on dist-test-slave-0b58"
server_key: "1a58929db89ef55a377e3c0d2f0b292e"
server_key_iv: "528d6fe6c937f848e299c6bd853c1891"
server_key_version: "encryptionkey@0"
I20260806 10:52:00.083076 1442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "85e5fb14afd54d068f0db84eb5e054b3"
format_stamp: "Formatted at 2026-08-06 10:52:00 on dist-test-slave-0b58"
server_key: "1a58929db89ef55a377e3c0d2f0b292e"
server_key_iv: "528d6fe6c937f848e299c6bd853c1891"
server_key_version: "encryptionkey@0"
I20260806 10:52:00.087792 1442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:52:00.091073 1457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:00.092347 1442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:00.092514 1442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "85e5fb14afd54d068f0db84eb5e054b3"
format_stamp: "Formatted at 2026-08-06 10:52:00 on dist-test-slave-0b58"
server_key: "1a58929db89ef55a377e3c0d2f0b292e"
server_key_iv: "528d6fe6c937f848e299c6bd853c1891"
server_key_version: "encryptionkey@0"
I20260806 10:52:00.092710 1442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:00.115866 1442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:00.119553 1442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:00.119841 1442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:00.120726 1442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:00.121923 1442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:00.122026 1442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:00.122169 1442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:00.122247 1442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:00.155738 1442 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:37721
I20260806 10:52:00.155817 1571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:37721 every 8 connection(s)
I20260806 10:52:00.157025 1442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:52:00.167456 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1442
I20260806 10:52:00.167668 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:52:00.168427 16468 external_mini_cluster.cc:1428] Setting key 3072b8b792b4df701d54162705210304
Aug 06 10:52:00 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013520, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:00.176524 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:00.159348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:50949 (local address 127.16.21.62:45285)
0806 10:52:00.159779 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:00.159783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:00.160744 (+ 961us) server_negotiation.cc:407] Connection header received
0806 10:52:00.162481 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:00.162486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:00.162571 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:00.162693 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:00.163983 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.164749 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.166298 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.166697 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.166835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:00.170529 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:00.170534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:00.170537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:00.170566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:00.173401 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:00.174286 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:00.174290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:00.174292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:00.174404 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:00.174880 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:00.174883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:00.174885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:00.175116 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:00.175219 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:00.176026 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 10:52:00.176180 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":211,"threads_started":1}
I20260806 10:52:00.177923 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:00.159874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45285 (local address 127.16.21.1:50949)
0806 10:52:00.160597 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:00.160633 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:00.162217 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:00.162910 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:00.162917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:00.163350 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:00.163828 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:00.163842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.164909 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.164913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:00.166137 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:00.166148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.166924 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.166932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:00.167160 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:00.167969 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:00.167990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:00.170320 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:00.173536 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:00.173547 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:00.173564 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:00.174172 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:00.174510 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:00.174517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:00.174524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:00.174777 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:00.175241 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:00.175251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:00.175654 (+ 403us) client_negotiation.cc:769] Sending connection context
0806 10:52:00.175948 (+ 294us) client_negotiation.cc:240] Negotiation successful
0806 10:52:00.176536 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":330,"threads_started":1}
I20260806 10:52:00.179481 1572 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45285
I20260806 10:52:00.179782 1572 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:00.180496 1572 heartbeater.cc:511] Master 127.16.21.62:45285 requested a full tablet report, sending...
I20260806 10:52:00.182489 1380 ts_manager.cc:194] Registered new tserver with Master: 85e5fb14afd54d068f0db84eb5e054b3 (127.16.21.1:37721)
I20260806 10:52:00.183908 1380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:50949
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:52:00.236754 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45285
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:00.558370 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:00.558629 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:00.558782 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:00.565474 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:00.565558 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:00.565680 1578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:52:00.574940 1578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:00.576054 1578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:00.577288 1578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:00.589879 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:00.589970 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:00.589931 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:00.590823 1578 server_base.cc:1034] running on GCE node
I20260806 10:52:00.591356 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:00.592242 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:00.593530 1578 hybrid_clock.cc:648] HybridClock initialized: now 1786013520593490 us; error 45 us; skew 500 ppm
I20260806 10:52:00.593845 1578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:00 dist-test-slave-0b58 krb5kdc[1334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013520, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:00.601532 1578 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:52:00.602927 1578 webserver.cc:460] Webserver started at http://127.16.21.2:36183/ using document root <none> and password file <none>
I20260806 10:52:00.603524 1578 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:00.603618 1578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:00.603861 1578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:00.606551 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "5074413e52544ccb98b81720d1733015"
format_stamp: "Formatted at 2026-08-06 10:52:00 on dist-test-slave-0b58"
server_key: "7313ad6abb476ccce14abeff44c1209d"
server_key_iv: "6feb2c5c96963f2d4259bd1f145a858f"
server_key_version: "encryptionkey@0"
I20260806 10:52:00.607242 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "5074413e52544ccb98b81720d1733015"
format_stamp: "Formatted at 2026-08-06 10:52:00 on dist-test-slave-0b58"
server_key: "7313ad6abb476ccce14abeff44c1209d"
server_key_iv: "6feb2c5c96963f2d4259bd1f145a858f"
server_key_version: "encryptionkey@0"
I20260806 10:52:00.611977 1578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260806 10:52:00.615124 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:00.616416 1578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:52:00.616572 1578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "5074413e52544ccb98b81720d1733015"
format_stamp: "Formatted at 2026-08-06 10:52:00 on dist-test-slave-0b58"
server_key: "7313ad6abb476ccce14abeff44c1209d"
server_key_iv: "6feb2c5c96963f2d4259bd1f145a858f"
server_key_version: "encryptionkey@0"
I20260806 10:52:00.616819 1578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:00.627862 1578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:00.631489 1578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:00.631773 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:00.632766 1578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:00.633977 1578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:00.634047 1578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:00.634236 1578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:00.634290 1578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:00.667069 1578 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:45827
I20260806 10:52:00.667120 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:45827 every 8 connection(s)
I20260806 10:52:00.668229 1578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:52:00.673184 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1578
I20260806 10:52:00.673395 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:52:00.673830 16468 external_mini_cluster.cc:1428] Setting key 59398740916d46e6cb6094d56eeb0ab7
Aug 06 10:52:00 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013520, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:00.687299 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:00.670625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:32829 (local address 127.16.21.62:45285)
0806 10:52:00.670796 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:00.670801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:00.671902 (+ 1101us) server_negotiation.cc:407] Connection header received
0806 10:52:00.673849 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:00.673854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:00.673943 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:00.674061 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:00.675656 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.676408 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.677911 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.678357 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.678484 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:00.681860 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:00.681864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:00.681867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:00.681897 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:00.684401 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:00.685092 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:00.685099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:00.685102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:00.685234 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:00.685641 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:00.685649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:00.685652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:00.685924 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:00.686091 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:00.686723 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 10:52:00.686943 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:52:00.688161 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:00.671024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45285 (local address 127.16.21.2:32829)
0806 10:52:00.671741 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:00.671776 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:00.673596 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:00.674278 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:00.674285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:00.674903 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:00.675501 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:00.675520 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.676543 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.676548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:00.677768 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:00.677784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:00.678567 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:00.678574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:00.678923 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:00.679674 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:00.679696 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:00.681648 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:00.684542 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:00.684550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:00.684565 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:00.684972 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:00.685344 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:00.685349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:00.685353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:00.685529 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:00.686051 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:00.686059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:00.686374 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 10:52:00.686627 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 10:52:00.687101 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":261,"threads_started":1}
I20260806 10:52:00.689584 1707 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45285
I20260806 10:52:00.689926 1707 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:00.690754 1707 heartbeater.cc:511] Master 127.16.21.62:45285 requested a full tablet report, sending...
I20260806 10:52:00.692246 1380 ts_manager.cc:194] Registered new tserver with Master: 5074413e52544ccb98b81720d1733015 (127.16.21.2:45827)
I20260806 10:52:00.693096 1380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:32829
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:52:00.741191 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45285
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:01.069949 1712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:01.070181 1712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:01.070283 1712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:01.077049 1712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:01.077147 1712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:01.077261 1712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:52:01.086189 1712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:01.087193 1712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:01.088387 1712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:01.100942 1717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:01.100909 1720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:01.100977 1718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:01.101969 1712 server_base.cc:1034] running on GCE node
I20260806 10:52:01.102595 1712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:01.103426 1712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:01.104645 1712 hybrid_clock.cc:648] HybridClock initialized: now 1786013521104561 us; error 74 us; skew 500 ppm
I20260806 10:52:01.104924 1712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:01 dist-test-slave-0b58 krb5kdc[1334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013521, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:01.107831 1712 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:52:01.109203 1712 webserver.cc:460] Webserver started at http://127.16.21.3:40221/ using document root <none> and password file <none>
I20260806 10:52:01.109855 1712 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:01.109997 1712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:01.110301 1712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:01.112987 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "d4ef230d9ebf4d819b45be67b3855ce1"
format_stamp: "Formatted at 2026-08-06 10:52:01 on dist-test-slave-0b58"
server_key: "10d6a8b464a36a015350a9e8180f2b5b"
server_key_iv: "407453d7877470cd5f42c0189fa7e717"
server_key_version: "encryptionkey@0"
I20260806 10:52:01.113693 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "d4ef230d9ebf4d819b45be67b3855ce1"
format_stamp: "Formatted at 2026-08-06 10:52:01 on dist-test-slave-0b58"
server_key: "10d6a8b464a36a015350a9e8180f2b5b"
server_key_iv: "407453d7877470cd5f42c0189fa7e717"
server_key_version: "encryptionkey@0"
I20260806 10:52:01.118508 1712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 10:52:01.121897 1727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:01.123071 1712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:52:01.123234 1712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "d4ef230d9ebf4d819b45be67b3855ce1"
format_stamp: "Formatted at 2026-08-06 10:52:01 on dist-test-slave-0b58"
server_key: "10d6a8b464a36a015350a9e8180f2b5b"
server_key_iv: "407453d7877470cd5f42c0189fa7e717"
server_key_version: "encryptionkey@0"
I20260806 10:52:01.123416 1712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:01.138542 1712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:01.142110 1712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:01.142390 1712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:01.143272 1712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:01.144529 1712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:01.144634 1712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:01.144783 1712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:01.144861 1712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:01.178421 1712 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:37063
I20260806 10:52:01.178462 1841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:37063 every 8 connection(s)
I20260806 10:52:01.179692 1712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:52:01.186655 1572 heartbeater.cc:503] Master 127.16.21.62:45285 was elected leader, sending a full tablet report...
I20260806 10:52:01.188710 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1712
I20260806 10:52:01.188917 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:52:01.189401 16468 external_mini_cluster.cc:1428] Setting key 3afc829e4e89402b797a83c232250171
Aug 06 10:52:01 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013521, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:01.199293 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.182160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:36675 (local address 127.16.21.62:45285)
0806 10:52:01.182367 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.182373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.183480 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 10:52:01.185361 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.185365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.185456 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.185542 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:01.187014 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.187962 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.189881 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.190309 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.190428 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.194056 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:01.194060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:01.194064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:01.194094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:01.196756 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.197442 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.197446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.197448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.197588 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.197948 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.197951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.197953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.198191 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:01.198311 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.198893 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.199044 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:52:01.200174 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.182635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45285 (local address 127.16.21.3:36675)
0806 10:52:01.183325 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:01.183369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:01.185114 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:01.185729 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:01.185738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:01.186223 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:01.186847 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.186867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.188127 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.188132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.189728 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.189742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.190533 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.190543 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.190817 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.191522 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:01.191543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:01.193875 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:01.196869 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:01.196875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:01.196884 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:01.197340 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:01.197689 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:01.197692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:01.197694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:01.197857 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:01.198297 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:01.198302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:01.198571 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 10:52:01.198798 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 10:52:01.199129 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":274,"threads_started":1}
I20260806 10:52:01.201900 1842 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45285
I20260806 10:52:01.202220 1842 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:01.203018 1842 heartbeater.cc:511] Master 127.16.21.62:45285 requested a full tablet report, sending...
I20260806 10:52:01.204782 1379 ts_manager.cc:194] Registered new tserver with Master: d4ef230d9ebf4d819b45be67b3855ce1 (127.16.21.3:37063)
I20260806 10:52:01.205606 1379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:36675
I20260806 10:52:01.216697 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:52:01.234311 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.221582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.16.21.62:45285)
0806 10:52:01.221733 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.221737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.222257 (+ 520us) server_negotiation.cc:407] Connection header received
0806 10:52:01.222508 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.222512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.222566 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.222664 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:01.224094 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.224830 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.226104 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.226630 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.226768 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.229348 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:01.229353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:01.229356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:01.229385 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:01.231678 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.232393 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.232399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.232403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.232546 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.232917 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.232922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.232925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.233212 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:01.233487 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.233808 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.234009 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:52:01.238011 1379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:52:01.245164 1379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:52:01.271476 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.256994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.16.21.3:37063)
0806 10:52:01.257816 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.257827 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.258694 (+ 867us) server_negotiation.cc:407] Connection header received
0806 10:52:01.258814 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.258825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.259114 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.259331 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:01.260444 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.262595 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.266232 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.269870 (+ 3638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.270007 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.270257 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.271009 (+ 752us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.271163 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":473,"threads_started":1}
I20260806 10:52:01.271500 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.256686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:37063 (local address 127.0.0.1:49616)
0806 10:52:01.258215 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:01.258265 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:01.258472 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:01.259692 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:01.259703 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:01.259744 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:01.260155 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.260179 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.262799 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.262811 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.265953 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.265974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.270389 (+ 4415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.270395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.270644 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.270708 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 10:52:01.270851 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 10:52:01.271050 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":7,"thread_start_us":2571,"threads_started":1}
I20260806 10:52:01.273638 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.257440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:45827 (local address 127.0.0.1:42056)
0806 10:52:01.259558 (+ 2118us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:01.259575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:01.259682 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:01.260995 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:01.261003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:01.261037 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:01.262881 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.262899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.267154 (+ 4255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.267163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.270743 (+ 3580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.270757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.272783 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.272790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.272971 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.273001 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:52:01.273102 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:52:01.273220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":104,"thread_start_us":436,"threads_started":1}
I20260806 10:52:01.273694 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.258126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.16.21.2:45827)
0806 10:52:01.260330 (+ 2204us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.260342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.260376 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:52:01.260491 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.260500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.260747 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.263169 (+ 2422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:01.263497 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.266962 (+ 3465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.270943 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.272572 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.272697 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.272855 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.273158 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.273344 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1989,"thread_start_us":1853,"threads_started":1}
I20260806 10:52:01.275681 1777 tablet_service.cc:1467] Processing CreateTablet for tablet c714277b9095493cad337ef58dafc5eb (DEFAULT_TABLE table=test-table [id=794ffb4d236d4feaa7eddb7d50218aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:01.277298 1777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c714277b9095493cad337ef58dafc5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:01.277746 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.260042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.16.21.1:37721)
0806 10:52:01.270155 (+ 10113us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.270163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.270188 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:52:01.270282 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.270290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.270487 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.270698 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:01.271341 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.273058 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.275629 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.276691 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.276838 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.277057 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.277270 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.277425 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9944,"thread_start_us":9797,"threads_started":1}
I20260806 10:52:01.277318 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c714277b9095493cad337ef58dafc5eb (DEFAULT_TABLE table=test-table [id=794ffb4d236d4feaa7eddb7d50218aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:01.277788 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.260330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37721 (local address 127.0.0.1:55870)
0806 10:52:01.269337 (+ 9007us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:01.269354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:01.269461 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:01.270677 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:01.270683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:01.270712 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:01.271151 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.271163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.273238 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.273243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.275482 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.275495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.276899 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.276905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.277081 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.277114 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:52:01.277227 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 10:52:01.277342 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8886,"mutex_wait_us":18,"thread_start_us":9199,"threads_started":1}
I20260806 10:52:01.278626 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c714277b9095493cad337ef58dafc5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:01.280722 1505 tablet_service.cc:1467] Processing CreateTablet for tablet c714277b9095493cad337ef58dafc5eb (DEFAULT_TABLE table=test-table [id=794ffb4d236d4feaa7eddb7d50218aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:01.281694 1505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c714277b9095493cad337ef58dafc5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:01.292927 1860 tablet_bootstrap.cc:492] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Bootstrap starting.
I20260806 10:52:01.294943 1861 tablet_bootstrap.cc:492] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Bootstrap starting.
I20260806 10:52:01.296165 1862 tablet_bootstrap.cc:492] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Bootstrap starting.
I20260806 10:52:01.296753 1860 tablet_bootstrap.cc:654] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:01.298128 1860 log.cc:824] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:01.299305 1861 tablet_bootstrap.cc:654] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:01.299542 1862 tablet_bootstrap.cc:654] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:01.300566 1861 log.cc:824] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:01.300769 1862 log.cc:824] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:01.301964 1860 tablet_bootstrap.cc:492] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: No bootstrap required, opened a new log
I20260806 10:52:01.302232 1860 ts_tablet_manager.cc:1397] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 10:52:01.303186 1861 tablet_bootstrap.cc:492] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: No bootstrap required, opened a new log
I20260806 10:52:01.303429 1861 ts_tablet_manager.cc:1397] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260806 10:52:01.305299 1862 tablet_bootstrap.cc:492] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: No bootstrap required, opened a new log
I20260806 10:52:01.305549 1862 ts_tablet_manager.cc:1397] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 10:52:01.309420 1860 raft_consensus.cc:348] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.309780 1860 raft_consensus.cc:374] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:01.309912 1860 raft_consensus.cc:729] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e5fb14afd54d068f0db84eb5e054b3, State: Initialized, Role: FOLLOWER
I20260806 10:52:01.310547 1860 consensus_queue.cc:260] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.312899 1860 ts_tablet_manager.cc:1428] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 10:52:01.314101 1861 raft_consensus.cc:348] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.314525 1861 raft_consensus.cc:374] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:01.314622 1861 raft_consensus.cc:729] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ef230d9ebf4d819b45be67b3855ce1, State: Initialized, Role: FOLLOWER
I20260806 10:52:01.315199 1861 consensus_queue.cc:260] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.316303 1842 heartbeater.cc:503] Master 127.16.21.62:45285 was elected leader, sending a full tablet report...
I20260806 10:52:01.316038 1862 raft_consensus.cc:348] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.316473 1862 raft_consensus.cc:374] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:01.316567 1862 raft_consensus.cc:729] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5074413e52544ccb98b81720d1733015, State: Initialized, Role: FOLLOWER
I20260806 10:52:01.316964 1861 ts_tablet_manager.cc:1428] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260806 10:52:01.317287 1862 consensus_queue.cc:260] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.318373 1707 heartbeater.cc:503] Master 127.16.21.62:45285 was elected leader, sending a full tablet report...
I20260806 10:52:01.319072 1862 ts_tablet_manager.cc:1428] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260806 10:52:01.374676 1868 raft_consensus.cc:482] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:52:01.375006 1868 raft_consensus.cc:504] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.376770 1868 leader_election.cc:288] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4ef230d9ebf4d819b45be67b3855ce1 (127.16.21.3:37063), 85e5fb14afd54d068f0db84eb5e054b3 (127.16.21.1:37721)
I20260806 10:52:01.384414 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.377207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:34237 (local address 127.16.21.3:37063)
0806 10:52:01.377472 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.377482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.377947 (+ 465us) server_negotiation.cc:407] Connection header received
0806 10:52:01.378223 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.378229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.378338 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.378449 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:01.379188 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.380160 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.382265 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.383485 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.383628 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.383748 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.384017 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.384139 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:52:01.384594 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.377099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:37063 (local address 127.16.21.2:34237)
0806 10:52:01.377834 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:01.377858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:01.378066 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:01.378552 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:01.378556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:01.378578 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:01.379028 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.379040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.380342 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.380348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.382106 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.382118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.383701 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.383707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.383854 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.383880 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:52:01.383991 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:52:01.384114 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":79,"thread_start_us":404,"threads_started":1}
I20260806 10:52:01.384624 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.378212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:44617 (local address 127.16.21.1:37721)
0806 10:52:01.378464 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.378469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.378792 (+ 323us) server_negotiation.cc:407] Connection header received
0806 10:52:01.378946 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.378952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.379038 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.379140 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:01.379718 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.380685 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.382597 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.383646 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.383803 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.383911 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.384194 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.384362 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 10:52:01.384714 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.377834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37721 (local address 127.16.21.2:44617)
0806 10:52:01.378681 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:01.378703 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:01.378811 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:01.379177 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:01.379181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:01.379204 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:01.379561 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.379575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.380830 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.380834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.382337 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:01.382360 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.383879 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.383885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:01.384046 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.384067 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 10:52:01.384168 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:52:01.384309 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":1,"thread_start_us":491,"threads_started":1}
I20260806 10:52:01.385843 1797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c714277b9095493cad337ef58dafc5eb" candidate_uuid: "5074413e52544ccb98b81720d1733015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" is_pre_election: true
I20260806 10:52:01.385840 1526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c714277b9095493cad337ef58dafc5eb" candidate_uuid: "5074413e52544ccb98b81720d1733015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e5fb14afd54d068f0db84eb5e054b3" is_pre_election: true
I20260806 10:52:01.386456 1797 raft_consensus.cc:2454] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5074413e52544ccb98b81720d1733015 in term 0.
I20260806 10:52:01.386456 1526 raft_consensus.cc:2454] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5074413e52544ccb98b81720d1733015 in term 0.
I20260806 10:52:01.387141 1597 leader_election.cc:302] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5074413e52544ccb98b81720d1733015, d4ef230d9ebf4d819b45be67b3855ce1; no voters:
I20260806 10:52:01.387468 1868 raft_consensus.cc:2781] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:52:01.387634 1868 raft_consensus.cc:482] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:52:01.387761 1868 raft_consensus.cc:3037] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:01.390020 1868 raft_consensus.cc:504] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.390568 1868 leader_election.cc:288] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [CANDIDATE]: Term 1 election: Requested vote from peers d4ef230d9ebf4d819b45be67b3855ce1 (127.16.21.3:37063), 85e5fb14afd54d068f0db84eb5e054b3 (127.16.21.1:37721)
I20260806 10:52:01.391040 1797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c714277b9095493cad337ef58dafc5eb" candidate_uuid: "5074413e52544ccb98b81720d1733015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ef230d9ebf4d819b45be67b3855ce1"
I20260806 10:52:01.391113 1526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c714277b9095493cad337ef58dafc5eb" candidate_uuid: "5074413e52544ccb98b81720d1733015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e5fb14afd54d068f0db84eb5e054b3"
I20260806 10:52:01.391263 1797 raft_consensus.cc:3037] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:01.391343 1526 raft_consensus.cc:3037] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:01.393608 1797 raft_consensus.cc:2454] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5074413e52544ccb98b81720d1733015 in term 1.
I20260806 10:52:01.393615 1526 raft_consensus.cc:2454] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5074413e52544ccb98b81720d1733015 in term 1.
I20260806 10:52:01.393993 1597 leader_election.cc:302] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5074413e52544ccb98b81720d1733015, 85e5fb14afd54d068f0db84eb5e054b3; no voters:
I20260806 10:52:01.394300 1868 raft_consensus.cc:2781] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:01.394730 1868 raft_consensus.cc:686] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 1 LEADER]: Becoming Leader. State: Replica: 5074413e52544ccb98b81720d1733015, State: Running, Role: LEADER
I20260806 10:52:01.395179 1868 consensus_queue.cc:237] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } }
I20260806 10:52:01.399977 1380 catalog_manager.cc:5495] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5074413e52544ccb98b81720d1733015 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "5074413e52544ccb98b81720d1733015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5074413e52544ccb98b81720d1733015" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 45827 } health_report { overall_health: HEALTHY } } }
W20260806 10:52:01.409228 1573 tablet.cc:2369] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:52:01.420225 1708 tablet.cc:2369] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:01.424580 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.419212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.16.21.2:45827)
0806 10:52:01.419387 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.419391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.419500 (+ 109us) server_negotiation.cc:407] Connection header received
0806 10:52:01.419708 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.419711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.419783 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.419900 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:52:01.420363 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.421194 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.422606 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.423000 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.423096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.423570 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:52:01.424030 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:52:01.424144 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.424241 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.424374 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 10:52:01.431408 1843 tablet.cc:2369] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:01.432538 1797 raft_consensus.cc:1261] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 1 FOLLOWER]: Refusing update from remote peer 5074413e52544ccb98b81720d1733015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:01.432552 1526 raft_consensus.cc:1261] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 1 FOLLOWER]: Refusing update from remote peer 5074413e52544ccb98b81720d1733015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:01.433436 1871 consensus_queue.cc:1035] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e5fb14afd54d068f0db84eb5e054b3" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:01.433806 1868 consensus_queue.cc:1035] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4ef230d9ebf4d819b45be67b3855ce1" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:01.445324 1876 mvcc.cc:205] Tried to move back new op lower bound from 7315511383777460224 to 7315511383638876160. Current Snapshot: MvccSnapshot[applied={T|T < 7315511383777460224}]
I20260806 10:52:01.458353 1379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:01.458638 1379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:01.462230 1379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e91055b6fbe34fa78b431d43b807d2a4: Sending DeleteTablet for 3 replicas of tablet c714277b9095493cad337ef58dafc5eb
I20260806 10:52:01.463451 1777 tablet_service.cc:1514] Processing DeleteTablet for tablet c714277b9095493cad337ef58dafc5eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:01 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:49616
I20260806 10:52:01.463603 1505 tablet_service.cc:1514] Processing DeleteTablet for tablet c714277b9095493cad337ef58dafc5eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:01 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:55870
I20260806 10:52:01.464124 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c714277b9095493cad337ef58dafc5eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:01 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:42056
I20260806 10:52:01.464745 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1442
I20260806 10:52:01.474475 1884 tablet_replica.cc:330] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: stopping tablet replica
I20260806 10:52:01.474888 1884 raft_consensus.cc:2229] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:01.475359 1884 raft_consensus.cc:2258] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:01.477617 1885 tablet_replica.cc:330] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: stopping tablet replica
I20260806 10:52:01.478019 1885 raft_consensus.cc:2229] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:01.478467 1885 raft_consensus.cc:2258] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:01.479434 1886 tablet_replica.cc:330] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: stopping tablet replica
I20260806 10:52:01.479799 1886 raft_consensus.cc:2229] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:52:01.480635 1885 ts_tablet_manager.cc:1905] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:01.481307 1884 ts_tablet_manager.cc:1905] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:01.485570 1886 raft_consensus.cc:2258] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:01.488235 1885 ts_tablet_manager.cc:1918] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:01.488296 1884 ts_tablet_manager.cc:1918] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:01.488444 1884 log.cc:1196] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/c714277b9095493cad337ef58dafc5eb
I20260806 10:52:01.488464 1885 log.cc:1196] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/c714277b9095493cad337ef58dafc5eb
Aug 06 10:52:01 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:52:01.488993 1885 ts_tablet_manager.cc:1939] T c714277b9095493cad337ef58dafc5eb P 85e5fb14afd54d068f0db84eb5e054b3: Deleting consensus metadata
I20260806 10:52:01.489065 1884 ts_tablet_manager.cc:1939] T c714277b9095493cad337ef58dafc5eb P d4ef230d9ebf4d819b45be67b3855ce1: Deleting consensus metadata
I20260806 10:52:01.490760 1366 catalog_manager.cc:4841] TS 85e5fb14afd54d068f0db84eb5e054b3 (127.16.21.1:37721): tablet c714277b9095493cad337ef58dafc5eb (table test-table [id=794ffb4d236d4feaa7eddb7d50218aa4]) successfully deleted
I20260806 10:52:01.491094 1366 catalog_manager.cc:4841] TS d4ef230d9ebf4d819b45be67b3855ce1 (127.16.21.3:37063): tablet c714277b9095493cad337ef58dafc5eb (table test-table [id=794ffb4d236d4feaa7eddb7d50218aa4]) successfully deleted
I20260806 10:52:01.491902 1886 ts_tablet_manager.cc:1905] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:01.497720 1886 ts_tablet_manager.cc:1918] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:01.497931 1886 log.cc:1196] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/c714277b9095493cad337ef58dafc5eb
I20260806 10:52:01.498564 1886 ts_tablet_manager.cc:1939] T c714277b9095493cad337ef58dafc5eb P 5074413e52544ccb98b81720d1733015: Deleting consensus metadata
I20260806 10:52:01.499065 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.466260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.16.21.1:37721)
0806 10:52:01.466459 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.466468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.476819 (+ 10351us) server_negotiation.cc:407] Connection header received
0806 10:52:01.477310 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.477320 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.477444 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.477533 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:01.480509 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.481577 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.483528 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.484157 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.484273 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.491522 (+ 7249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:01.491535 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:01.491551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:01.491596 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:01.495462 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.496411 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.496420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.496424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.496589 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.497098 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.497104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.497108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.497537 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:01.497925 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.498225 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.498474 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:52:01.499701 1567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:01.500241 1365 catalog_manager.cc:4841] TS 5074413e52544ccb98b81720d1733015 (127.16.21.2:45827): tablet c714277b9095493cad337ef58dafc5eb (table test-table [id=794ffb4d236d4feaa7eddb7d50218aa4]) successfully deleted
I20260806 10:52:01.599939 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1442
I20260806 10:52:01.621628 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1578
Aug 06 10:52:01 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:52:01.638463 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.622454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.16.21.2:45827)
0806 10:52:01.622616 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.622622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.622722 (+ 100us) server_negotiation.cc:407] Connection header received
0806 10:52:01.622940 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.622944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.623009 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.623123 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:01.624529 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.625319 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.626728 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.627150 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.627309 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.631549 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:01.631565 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:01.631585 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:01.631643 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:01.635578 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.636220 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.636232 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.636240 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.636422 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.636859 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.636864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.636867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.637317 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:01.637681 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.637855 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.638075 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:52:01.639273 1702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:01.754401 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1578
I20260806 10:52:01.776140 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1712
Aug 06 10:52:01 dist-test-slave-0b58 krb5kdc[1334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:52:01.790923 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:01.776936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.16.21.3:37063)
0806 10:52:01.777122 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:01.777127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:01.777144 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:52:01.777310 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:01.777314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:01.777379 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:01.777460 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:01.778691 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.779526 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.780905 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:01.781283 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:01.781376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:01.785576 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:01.785590 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:01.785604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:01.785641 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:01.788115 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.788830 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.788837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.788839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.788986 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:01.789367 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:01.789376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:01.789378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:01.789734 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:01.790039 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:01.790366 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 10:52:01.790592 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:52:01.791586 1837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:01.880844 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1712
I20260806 10:52:01.900764 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1347
I20260806 10:52:01.901958 1411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:02.012926 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1347
2026-08-06T10:52:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4349 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T10:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:02Z Disabled control of system clock
I20260806 10:52:02.107535 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45149
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:52:02.445655 1896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:02.445957 1896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:02.446105 1896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:02.453181 1896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:02.453320 1896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:02.453392 1896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:02.453457 1896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:02.453502 1896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:02.462200 1896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:45149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:02.463338 1896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:02.465148 1896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:02.476691 1901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:02.476663 1902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:02.476974 1904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:02.477660 1896 server_base.cc:1034] running on GCE node
I20260806 10:52:02.478335 1896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:02.479581 1896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:02.480778 1896 hybrid_clock.cc:648] HybridClock initialized: now 1786013522480739 us; error 32 us; skew 500 ppm
I20260806 10:52:02.481130 1896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:02.483284 1896 webserver.cc:460] Webserver started at http://127.16.21.62:39915/ using document root <none> and password file <none>
I20260806 10:52:02.483943 1896 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:02.484055 1896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:02.484458 1896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:02.487248 1896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "ad5961d32549491f944beef69dcd41d9"
format_stamp: "Formatted at 2026-08-06 10:52:02 on dist-test-slave-0b58"
server_key: "ec3e332d71e8ea45f3a1d41711b1193d"
server_key_iv: "84fbd0a1a1415c761ce2e9f5761238b9"
server_key_version: "encryptionkey@0"
I20260806 10:52:02.488034 1896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "ad5961d32549491f944beef69dcd41d9"
format_stamp: "Formatted at 2026-08-06 10:52:02 on dist-test-slave-0b58"
server_key: "ec3e332d71e8ea45f3a1d41711b1193d"
server_key_iv: "84fbd0a1a1415c761ce2e9f5761238b9"
server_key_version: "encryptionkey@0"
I20260806 10:52:02.492846 1896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:52:02.496018 1910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:02.497267 1896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 10:52:02.497453 1896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "ad5961d32549491f944beef69dcd41d9"
format_stamp: "Formatted at 2026-08-06 10:52:02 on dist-test-slave-0b58"
server_key: "ec3e332d71e8ea45f3a1d41711b1193d"
server_key_iv: "84fbd0a1a1415c761ce2e9f5761238b9"
server_key_version: "encryptionkey@0"
I20260806 10:52:02.497663 1896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:02.511082 1896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:02.511843 1896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:02.512125 1896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:02.531122 1896 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:45149
I20260806 10:52:02.531164 1961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:45149 every 8 connection(s)
I20260806 10:52:02.532445 1896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:02.534786 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1896
I20260806 10:52:02.534987 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:02.535445 16468 external_mini_cluster.cc:1428] Setting key c61419075bc2c06fd98bfe3d3b9b3317
I20260806 10:52:02.536091 1962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:02.550565 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:02.537273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.16.21.62:45149)
0806 10:52:02.538357 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:02.538373 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:02.538419 (+ 46us) server_negotiation.cc:407] Connection header received
0806 10:52:02.540778 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:02.540803 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:02.541374 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:02.541779 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:02.542294 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:02.544334 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:02.545627 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:02.546667 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:02.546852 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:02.547953 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:02.547963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:02.547981 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:02.548710 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:02.548818 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:02.548848 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:02.548960 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:02.549151 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 10:52:02.549430 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":358,"threads_started":1}
I20260806 10:52:02.551003 1962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9: Bootstrap starting.
I20260806 10:52:02.554018 1962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:02.554966 1962 log.cc:824] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:02.557484 1962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9: No bootstrap required, opened a new log
I20260806 10:52:02.564949 1962 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad5961d32549491f944beef69dcd41d9" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45149 } }
I20260806 10:52:02.565289 1962 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:02.565438 1962 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad5961d32549491f944beef69dcd41d9, State: Initialized, Role: FOLLOWER
I20260806 10:52:02.565950 1962 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad5961d32549491f944beef69dcd41d9" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45149 } }
I20260806 10:52:02.566143 1962 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:02.566253 1962 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:02.566377 1962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:02.568450 1962 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad5961d32549491f944beef69dcd41d9" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45149 } }
I20260806 10:52:02.568933 1962 leader_election.cc:302] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad5961d32549491f944beef69dcd41d9; no voters:
I20260806 10:52:02.569636 1962 leader_election.cc:288] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:02.569753 1967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:02.570477 1967 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [term 1 LEADER]: Becoming Leader. State: Replica: ad5961d32549491f944beef69dcd41d9, State: Running, Role: LEADER
I20260806 10:52:02.571066 1967 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad5961d32549491f944beef69dcd41d9" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45149 } }
I20260806 10:52:02.571825 1962 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:02.573982 1968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad5961d32549491f944beef69dcd41d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad5961d32549491f944beef69dcd41d9" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45149 } } }
I20260806 10:52:02.574106 1969 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad5961d32549491f944beef69dcd41d9. Latest consensus state: current_term: 1 leader_uuid: "ad5961d32549491f944beef69dcd41d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad5961d32549491f944beef69dcd41d9" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 45149 } } }
I20260806 10:52:02.574399 1968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:02.574482 1969 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:02.575845 1973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:02.581089 1973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:02.590528 1973 catalog_manager.cc:1347] Generated new cluster ID: ba4d224d828746b2a4dfeb92b2fb1a79
I20260806 10:52:02.590654 1973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:02.620961 1973 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:02.622298 1973 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:02.644972 1973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9: Generated new TSK 0
I20260806 10:52:02.645792 1973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:02.659307 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45149
--builtin_ntp_servers=127.16.21.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:52:03.016680 1986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:03.016940 1986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:03.017112 1986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:03.024304 1986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:03.024434 1986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:03.024603 1986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:52:03.033382 1986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:03.034574 1986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:03.036244 1986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:03.049122 1994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:03.049261 1991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:03.049226 1992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:03.049804 1986 server_base.cc:1034] running on GCE node
I20260806 10:52:03.050379 1986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:03.051401 1986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:03.052629 1986 hybrid_clock.cc:648] HybridClock initialized: now 1786013523052581 us; error 43 us; skew 500 ppm
I20260806 10:52:03.053025 1986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:03.055151 1986 webserver.cc:460] Webserver started at http://127.16.21.1:40123/ using document root <none> and password file <none>
I20260806 10:52:03.055732 1986 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:03.055825 1986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:03.056073 1986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:03.058940 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "9f9c33ee75fc4cd1919b5fa6a880107c"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "6851bb814a65deecd088f4c7ca0b2603"
server_key_iv: "e895ee7a11e4b17a359156618ed6d061"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.059725 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "9f9c33ee75fc4cd1919b5fa6a880107c"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "6851bb814a65deecd088f4c7ca0b2603"
server_key_iv: "e895ee7a11e4b17a359156618ed6d061"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.064424 1986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 10:52:03.067669 2000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.068862 1986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:52:03.069033 1986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "9f9c33ee75fc4cd1919b5fa6a880107c"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "6851bb814a65deecd088f4c7ca0b2603"
server_key_iv: "e895ee7a11e4b17a359156618ed6d061"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.069242 1986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:03.082971 1986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:03.083709 1986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:03.084019 1986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:03.085038 1986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:03.086392 1986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:03.086514 1986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.086647 1986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:03.086740 1986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.119830 1986 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:40905
I20260806 10:52:03.119889 2115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:40905 every 8 connection(s)
I20260806 10:52:03.121212 1986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:52:03.127796 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 1986
I20260806 10:52:03.127982 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:52:03.128469 16468 external_mini_cluster.cc:1428] Setting key 427b91ab604ff4c6faa2deede0210c29
I20260806 10:52:03.131455 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45149
--builtin_ntp_servers=127.16.21.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:52:03.134563 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:03.123784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:43215 (local address 127.16.21.62:45149)
0806 10:52:03.124228 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:03.124232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:03.125417 (+ 1185us) server_negotiation.cc:407] Connection header received
0806 10:52:03.127222 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:03.127228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:03.127321 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:03.127437 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:03.128599 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.129537 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.131736 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.132146 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.132240 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:03.133856 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:03.133860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:03.133863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:03.133880 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:03.133920 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:03.133932 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:03.133997 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:03.134245 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 10:52:03.134327 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260806 10:52:03.135931 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:03.124423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45149 (local address 127.16.21.1:43215)
0806 10:52:03.125273 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:03.125307 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:03.126962 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:03.127669 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:03.127677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:03.127758 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:03.128438 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:03.128456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.129698 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.129703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:03.131574 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:03.131593 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.132394 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.132404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:03.132732 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:03.133499 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:03.133504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:03.133558 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:03.133657 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:03.134027 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:03.134041 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:03.134051 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:52:03.134217 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 10:52:03.134604 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":261,"threads_started":1}
I20260806 10:52:03.137419 2116 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45149
I20260806 10:52:03.137696 2116 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:03.138407 2116 heartbeater.cc:511] Master 127.16.21.62:45149 requested a full tablet report, sending...
I20260806 10:52:03.140339 1927 ts_manager.cc:194] Registered new tserver with Master: 9f9c33ee75fc4cd1919b5fa6a880107c (127.16.21.1:40905)
I20260806 10:52:03.141779 1927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.1:43215
W20260806 10:52:03.465130 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:03.465392 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:03.465471 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:03.472672 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:03.472759 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:03.472965 2120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:52:03.481922 2120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:03.483101 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:03.484925 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:03.497254 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:03.497332 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:03.497361 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:03.498330 2120 server_base.cc:1034] running on GCE node
I20260806 10:52:03.498878 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:03.499724 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:03.500962 2120 hybrid_clock.cc:648] HybridClock initialized: now 1786013523500908 us; error 62 us; skew 500 ppm
I20260806 10:52:03.501267 2120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:03.503441 2120 webserver.cc:460] Webserver started at http://127.16.21.2:39987/ using document root <none> and password file <none>
I20260806 10:52:03.504069 2120 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:03.504194 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:03.504547 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:03.507191 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "27f780690a014a5dade49c879212fecb"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "226e087cabd87539696c63a6a332a34d"
server_key_iv: "73a529537f1ed6cd9ba3cf70ab7e03e2"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.507999 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "27f780690a014a5dade49c879212fecb"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "226e087cabd87539696c63a6a332a34d"
server_key_iv: "73a529537f1ed6cd9ba3cf70ab7e03e2"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.512905 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 10:52:03.516201 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.517434 2120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:52:03.517607 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "27f780690a014a5dade49c879212fecb"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "226e087cabd87539696c63a6a332a34d"
server_key_iv: "73a529537f1ed6cd9ba3cf70ab7e03e2"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.517809 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:03.551352 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:03.552104 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:03.552428 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:03.553399 2120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:03.554714 2120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:03.554848 2120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.554967 2120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:03.555052 2120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.587990 2120 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:38043
I20260806 10:52:03.588068 2249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:38043 every 8 connection(s)
I20260806 10:52:03.589269 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:52:03.594861 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2120
I20260806 10:52:03.595044 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:52:03.595642 16468 external_mini_cluster.cc:1428] Setting key 0844225681f25f134346498c89188967
I20260806 10:52:03.598871 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45149
--builtin_ntp_servers=127.16.21.20:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:52:03.602908 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:03.591674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:40715 (local address 127.16.21.62:45149)
0806 10:52:03.591847 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:03.591851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:03.593104 (+ 1253us) server_negotiation.cc:407] Connection header received
0806 10:52:03.595042 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:03.595049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:03.595160 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:03.595252 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:03.596586 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.597658 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.600015 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.600450 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.600542 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:03.602216 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:03.602221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:03.602224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:03.602240 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:03.602275 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:03.602287 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:03.602353 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:03.602615 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 10:52:03.602685 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:52:03.604312 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:03.592082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45149 (local address 127.16.21.2:40715)
0806 10:52:03.592959 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:03.593001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:03.594723 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:03.595520 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:03.595530 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:03.595630 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:03.596398 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:03.596417 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.597830 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.597835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:03.599850 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:03.599869 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:03.600740 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:03.600751 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:03.601093 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:03.601899 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:03.601905 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:03.601952 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:03.602047 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:03.602381 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:03.602394 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:03.602405 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 10:52:03.602601 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 10:52:03.602973 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":307,"threads_started":1}
I20260806 10:52:03.605664 2250 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45149
I20260806 10:52:03.605938 2250 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:03.606631 2250 heartbeater.cc:511] Master 127.16.21.62:45149 requested a full tablet report, sending...
I20260806 10:52:03.607966 1927 ts_manager.cc:194] Registered new tserver with Master: 27f780690a014a5dade49c879212fecb (127.16.21.2:38043)
I20260806 10:52:03.608814 1927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.2:40715
W20260806 10:52:03.932152 2253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:03.932459 2253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:03.932653 2253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:03.939659 2253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:03.939744 2253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:03.939927 2253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:52:03.948544 2253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:45149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:03.949723 2253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:03.951396 2253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:03.963693 2258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:03.963768 2259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:03.963800 2261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:03.964851 2253 server_base.cc:1034] running on GCE node
I20260806 10:52:03.965493 2253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:03.966408 2253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:03.967631 2253 hybrid_clock.cc:648] HybridClock initialized: now 1786013523967575 us; error 46 us; skew 500 ppm
I20260806 10:52:03.967943 2253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:03.970163 2253 webserver.cc:460] Webserver started at http://127.16.21.3:33605/ using document root <none> and password file <none>
I20260806 10:52:03.970829 2253 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:03.970954 2253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:03.971278 2253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:03.974325 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "d3d50f7a21224df69bc5ee8407e753ba"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "9cae108bccefc330a40930dd11773694"
server_key_iv: "95b5343cb29f002b53430ad2024179ea"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.975085 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d50f7a21224df69bc5ee8407e753ba"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "9cae108bccefc330a40930dd11773694"
server_key_iv: "95b5343cb29f002b53430ad2024179ea"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.979773 2253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260806 10:52:03.983397 2268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:03.984725 2253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:03.984899 2253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "d3d50f7a21224df69bc5ee8407e753ba"
format_stamp: "Formatted at 2026-08-06 10:52:03 on dist-test-slave-0b58"
server_key: "9cae108bccefc330a40930dd11773694"
server_key_iv: "95b5343cb29f002b53430ad2024179ea"
server_key_version: "encryptionkey@0"
I20260806 10:52:03.985110 2253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:04.007805 2253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:04.008797 2253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:04.009093 2253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:04.010054 2253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:04.011473 2253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:04.011600 2253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:04.011746 2253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:04.011838 2253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:04.045331 2253 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:40415
I20260806 10:52:04.045414 2383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:40415 every 8 connection(s)
I20260806 10:52:04.046626 2253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:52:04.051726 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2253
I20260806 10:52:04.051921 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:52:04.052567 16468 external_mini_cluster.cc:1428] Setting key b6843aa1e6c5e91a8e231af73b5d1cbe
I20260806 10:52:04.059741 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.049157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:37501 (local address 127.16.21.62:45149)
0806 10:52:04.049401 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.049407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.050730 (+ 1323us) server_negotiation.cc:407] Connection header received
0806 10:52:04.053119 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.053125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.053219 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.053338 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:04.054581 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.055745 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.057314 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.057794 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.057889 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.059090 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:04.059095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:04.059098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:04.059114 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:04.059145 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:04.059157 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:04.059233 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.059428 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.059501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 10:52:04.060714 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.049696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:45149 (local address 127.16.21.3:37501)
0806 10:52:04.050572 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.050611 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:04.052819 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:04.053575 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:04.053591 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:04.053680 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:04.054348 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.054374 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.055893 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.055897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.057180 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.057193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.057975 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.057980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.058170 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.058809 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:04.058816 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:04.058851 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:04.058925 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:04.059246 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:04.059254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:04.059259 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:52:04.059399 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 10:52:04.059659 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
I20260806 10:52:04.062049 2384 heartbeater.cc:347] Connected to a master server at 127.16.21.62:45149
I20260806 10:52:04.062333 2384 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:04.063024 2384 heartbeater.cc:511] Master 127.16.21.62:45149 requested a full tablet report, sending...
I20260806 10:52:04.064431 1927 ts_manager.cc:194] Registered new tserver with Master: d3d50f7a21224df69bc5ee8407e753ba (127.16.21.3:40415)
I20260806 10:52:04.065181 1927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.3:37501
I20260806 10:52:04.067744 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:52:04.078007 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.071802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.16.21.62:45149)
0806 10:52:04.072004 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.072012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.072556 (+ 544us) server_negotiation.cc:407] Connection header received
0806 10:52:04.072771 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.072789 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.072854 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.072952 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:04.073444 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.074339 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.075672 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.076085 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.076216 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.077301 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:04.077306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:04.077310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:04.077325 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:04.077356 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:04.077368 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:04.077469 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.077625 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.077723 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:52:04.081573 1927 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:52:04.088531 1927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:52:04.110270 2396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.100064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:38043 (local address 127.0.0.1:47082)
0806 10:52:04.101075 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.101127 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:04.101295 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:04.103257 (+ 1962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:04.103268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:04.103331 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:04.103813 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.103826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.105953 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.105962 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.107956 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.107973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.109324 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.109330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.109542 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.109608 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 10:52:04.109758 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 10:52:04.109952 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":649,"threads_started":1}
I20260806 10:52:04.110322 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.100620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.16.21.2:38043)
0806 10:52:04.102484 (+ 1864us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.102496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.102533 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:52:04.102652 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.102664 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.102953 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.104043 (+ 1090us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:04.104400 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.105819 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.108157 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.109127 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.109294 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.109536 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.109800 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.109981 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":1503,"threads_started":1}
I20260806 10:52:04.112409 2398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.100102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41168)
0806 10:52:04.103455 (+ 3353us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.103474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:04.103620 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:04.104557 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:04.104564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:04.104592 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:04.105100 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.105119 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.107239 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.107246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.109849 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.109865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.111578 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.111584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.111750 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.111788 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:52:04.111892 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 10:52:04.112011 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3188,"mutex_wait_us":116,"spinlock_wait_cycles":6912,"thread_start_us":3382,"threads_started":1}
I20260806 10:52:04.112490 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.101049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.16.21.1:40905)
0806 10:52:04.101970 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.101982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.103761 (+ 1779us) server_negotiation.cc:407] Connection header received
0806 10:52:04.103871 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.103878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.104100 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.104296 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:04.105486 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.107088 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.110098 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.111366 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.111514 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.111748 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.111959 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.112115 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":568,"threads_started":1}
I20260806 10:52:04.113682 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.100573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:40415 (local address 127.0.0.1:57306)
0806 10:52:04.103778 (+ 3205us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.103805 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:04.103918 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:04.104625 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:04.104629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:04.104661 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:04.105150 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.105164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.107143 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.107150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.109849 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.109865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.112939 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.112958 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.113115 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.113145 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:52:04.113239 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 10:52:04.113366 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3064,"mutex_wait_us":177,"thread_start_us":2568,"threads_started":1}
I20260806 10:52:04.113876 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.101242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.16.21.3:40415)
0806 10:52:04.102936 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.102947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.104032 (+ 1085us) server_negotiation.cc:407] Connection header received
0806 10:52:04.104122 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.104132 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.104365 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.104615 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:04.105341 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.106965 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.110947 (+ 3982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.112364 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.112503 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.112757 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.113298 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.113484 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1496,"thread_start_us":1340,"threads_started":1}
I20260806 10:52:04.114513 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 8f64a41c64c540d68241916c40902e85 (DEFAULT_TABLE table=test-table [id=87d7719c9524428d8e1630b044d95446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:04.115971 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f64a41c64c540d68241916c40902e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:04.116014 2050 tablet_service.cc:1467] Processing CreateTablet for tablet 8f64a41c64c540d68241916c40902e85 (DEFAULT_TABLE table=test-table [id=87d7719c9524428d8e1630b044d95446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:04.117363 2050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f64a41c64c540d68241916c40902e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:04.117386 2318 tablet_service.cc:1467] Processing CreateTablet for tablet 8f64a41c64c540d68241916c40902e85 (DEFAULT_TABLE table=test-table [id=87d7719c9524428d8e1630b044d95446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:04.118788 2318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f64a41c64c540d68241916c40902e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:04.134352 2402 tablet_bootstrap.cc:492] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Bootstrap starting.
I20260806 10:52:04.137872 2402 tablet_bootstrap.cc:654] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:04.138022 2403 tablet_bootstrap.cc:492] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Bootstrap starting.
I20260806 10:52:04.139221 2402 log.cc:824] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:04.140714 2404 tablet_bootstrap.cc:492] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c: Bootstrap starting.
I20260806 10:52:04.141700 2403 tablet_bootstrap.cc:654] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:04.142304 2402 tablet_bootstrap.cc:492] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: No bootstrap required, opened a new log
I20260806 10:52:04.142584 2402 ts_tablet_manager.cc:1397] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:52:04.142930 2403 log.cc:824] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:04.144129 2404 tablet_bootstrap.cc:654] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:04.144649 2116 heartbeater.cc:503] Master 127.16.21.62:45149 was elected leader, sending a full tablet report...
I20260806 10:52:04.145457 2404 log.cc:824] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:04.146598 2403 tablet_bootstrap.cc:492] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: No bootstrap required, opened a new log
I20260806 10:52:04.146886 2403 ts_tablet_manager.cc:1397] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 10:52:04.151804 2402 raft_consensus.cc:348] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.152170 2402 raft_consensus.cc:374] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:04.152323 2402 raft_consensus.cc:729] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f780690a014a5dade49c879212fecb, State: Initialized, Role: FOLLOWER
I20260806 10:52:04.152992 2402 consensus_queue.cc:260] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.154130 2250 heartbeater.cc:503] Master 127.16.21.62:45149 was elected leader, sending a full tablet report...
I20260806 10:52:04.154575 2404 tablet_bootstrap.cc:492] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c: No bootstrap required, opened a new log
I20260806 10:52:04.154839 2402 ts_tablet_manager.cc:1428] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 10:52:04.154839 2404 ts_tablet_manager.cc:1397] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260806 10:52:04.158298 2403 raft_consensus.cc:348] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.158756 2403 raft_consensus.cc:374] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:04.158881 2403 raft_consensus.cc:729] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d50f7a21224df69bc5ee8407e753ba, State: Initialized, Role: FOLLOWER
I20260806 10:52:04.159515 2403 consensus_queue.cc:260] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.160588 2384 heartbeater.cc:503] Master 127.16.21.62:45149 was elected leader, sending a full tablet report...
I20260806 10:52:04.161038 2403 ts_tablet_manager.cc:1428] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 10:52:04.164579 2404 raft_consensus.cc:348] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.165046 2404 raft_consensus.cc:374] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:04.165163 2404 raft_consensus.cc:729] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9c33ee75fc4cd1919b5fa6a880107c, State: Initialized, Role: FOLLOWER
I20260806 10:52:04.165700 2404 consensus_queue.cc:260] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.167095 2404 ts_tablet_manager.cc:1428] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 10:52:04.210032 2410 raft_consensus.cc:482] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:52:04.210292 2410 raft_consensus.cc:504] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.212447 2410 leader_election.cc:288] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f780690a014a5dade49c879212fecb (127.16.21.2:38043), 9f9c33ee75fc4cd1919b5fa6a880107c (127.16.21.1:40905)
I20260806 10:52:04.220379 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.212627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:38043 (local address 127.16.21.3:39395)
0806 10:52:04.212898 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.212920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:04.213062 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:04.214072 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:04.214080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:04.214104 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:04.214293 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.214305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.215745 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.215750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.217858 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.217886 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.219595 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.219601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.219776 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.219811 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:52:04.219901 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 10:52:04.220036 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 10:52:04.220947 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.213346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:39395 (local address 127.16.21.2:38043)
0806 10:52:04.213551 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.213560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.213583 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:52:04.213675 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.213680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.213770 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.213868 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:04.214481 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.215573 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.218055 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.219298 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.219412 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.219508 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.220567 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.220696 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:52:04.221426 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.213338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:38401 (local address 127.16.21.1:40905)
0806 10:52:04.214311 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.214316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.214336 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:52:04.214408 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.214415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.214500 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.214584 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:04.215346 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.216411 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.218806 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.220000 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.220087 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.220176 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.221020 (+ 844us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.221175 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818}
I20260806 10:52:04.221643 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.213121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.16.21.3:38401)
0806 10:52:04.213914 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.213932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:04.214031 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:04.214732 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:04.214737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:04.214760 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:04.215130 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.215141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.216625 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.216636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.218584 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:04.218597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.220349 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.220356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:04.220822 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.220857 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:52:04.220951 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 10:52:04.221264 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":420,"threads_started":1}
I20260806 10:52:04.222486 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f64a41c64c540d68241916c40902e85" candidate_uuid: "d3d50f7a21224df69bc5ee8407e753ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f780690a014a5dade49c879212fecb" is_pre_election: true
I20260806 10:52:04.222822 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f64a41c64c540d68241916c40902e85" candidate_uuid: "d3d50f7a21224df69bc5ee8407e753ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" is_pre_election: true
I20260806 10:52:04.223044 2204 raft_consensus.cc:2454] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d50f7a21224df69bc5ee8407e753ba in term 0.
I20260806 10:52:04.223354 2070 raft_consensus.cc:2454] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d50f7a21224df69bc5ee8407e753ba in term 0.
I20260806 10:52:04.223860 2271 leader_election.cc:302] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f780690a014a5dade49c879212fecb, d3d50f7a21224df69bc5ee8407e753ba; no voters:
I20260806 10:52:04.224337 2410 raft_consensus.cc:2781] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:52:04.224524 2410 raft_consensus.cc:482] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:52:04.224638 2410 raft_consensus.cc:3037] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:04.228284 2410 raft_consensus.cc:504] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.229025 2410 leader_election.cc:288] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [CANDIDATE]: Term 1 election: Requested vote from peers 27f780690a014a5dade49c879212fecb (127.16.21.2:38043), 9f9c33ee75fc4cd1919b5fa6a880107c (127.16.21.1:40905)
I20260806 10:52:04.229511 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f64a41c64c540d68241916c40902e85" candidate_uuid: "d3d50f7a21224df69bc5ee8407e753ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f780690a014a5dade49c879212fecb"
I20260806 10:52:04.229521 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f64a41c64c540d68241916c40902e85" candidate_uuid: "d3d50f7a21224df69bc5ee8407e753ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c"
I20260806 10:52:04.229729 2204 raft_consensus.cc:3037] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:04.229734 2070 raft_consensus.cc:3037] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:04.232806 2204 raft_consensus.cc:2454] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d50f7a21224df69bc5ee8407e753ba in term 1.
I20260806 10:52:04.232880 2070 raft_consensus.cc:2454] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d50f7a21224df69bc5ee8407e753ba in term 1.
I20260806 10:52:04.233318 2270 leader_election.cc:302] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9c33ee75fc4cd1919b5fa6a880107c, d3d50f7a21224df69bc5ee8407e753ba; no voters:
I20260806 10:52:04.233681 2410 raft_consensus.cc:2781] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:04.234138 2410 raft_consensus.cc:686] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 1 LEADER]: Becoming Leader. State: Replica: d3d50f7a21224df69bc5ee8407e753ba, State: Running, Role: LEADER
I20260806 10:52:04.234593 2410 consensus_queue.cc:237] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } }
I20260806 10:52:04.239199 1927 catalog_manager.cc:5495] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba reported cstate change: term changed from 0 to 1, leader changed from <none> to d3d50f7a21224df69bc5ee8407e753ba (127.16.21.3). New cstate: current_term: 1 leader_uuid: "d3d50f7a21224df69bc5ee8407e753ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d50f7a21224df69bc5ee8407e753ba" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 40415 } health_report { overall_health: HEALTHY } } }
I20260806 10:52:04.271131 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.265935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.16.21.3:40415)
0806 10:52:04.266113 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.266117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.266132 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:52:04.266223 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.266227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.266304 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.266419 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:52:04.266746 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.267435 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.268959 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.269407 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.269537 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.270040 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:52:04.270571 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:52:04.270724 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.270824 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.270901 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:52:04.279173 2204 raft_consensus.cc:1261] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 1 FOLLOWER]: Refusing update from remote peer d3d50f7a21224df69bc5ee8407e753ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:04.279172 2070 raft_consensus.cc:1261] T 8f64a41c64c540d68241916c40902e85 P 9f9c33ee75fc4cd1919b5fa6a880107c [term 1 FOLLOWER]: Refusing update from remote peer d3d50f7a21224df69bc5ee8407e753ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:04.280115 2413 consensus_queue.cc:1035] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9c33ee75fc4cd1919b5fa6a880107c" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:04.280517 2410 consensus_queue.cc:1035] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f780690a014a5dade49c879212fecb" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 38043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:04.291266 2416 mvcc.cc:205] Tried to move back new op lower bound from 7315511395437690880 to 7315511395268734976. Current Snapshot: MvccSnapshot[applied={T|T < 7315511395437690880}]
I20260806 10:52:04.304993 1926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:04.305260 1926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:04.308897 1926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad5961d32549491f944beef69dcd41d9: Sending DeleteTablet for 3 replicas of tablet 8f64a41c64c540d68241916c40902e85
I20260806 10:52:04.309947 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f64a41c64c540d68241916c40902e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:04 UTC) from {username='slave'} at 127.0.0.1:47082
I20260806 10:52:04.310012 2050 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f64a41c64c540d68241916c40902e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:04 UTC) from {username='slave'} at 127.0.0.1:41168
I20260806 10:52:04.310669 2318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f64a41c64c540d68241916c40902e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:04 UTC) from {username='slave'} at 127.0.0.1:57306
I20260806 10:52:04.310863 2425 tablet_replica.cc:330] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: stopping tablet replica
I20260806 10:52:04.311254 2425 raft_consensus.cc:2229] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:04.311916 2425 raft_consensus.cc:2258] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:04.314172 2425 ts_tablet_manager.cc:1905] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:04.316502 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1986
I20260806 10:52:04.322080 2425 ts_tablet_manager.cc:1918] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:04.322345 2425 log.cc:1196] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/8f64a41c64c540d68241916c40902e85
I20260806 10:52:04.323024 2425 ts_tablet_manager.cc:1939] T 8f64a41c64c540d68241916c40902e85 P 27f780690a014a5dade49c879212fecb: Deleting consensus metadata
I20260806 10:52:04.325037 1913 catalog_manager.cc:4841] TS 27f780690a014a5dade49c879212fecb (127.16.21.2:38043): tablet 8f64a41c64c540d68241916c40902e85 (table test-table [id=87d7719c9524428d8e1630b044d95446]) successfully deleted
I20260806 10:52:04.329523 2427 tablet_replica.cc:330] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: stopping tablet replica
I20260806 10:52:04.330032 2427 raft_consensus.cc:2229] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:52:04.330823 2427 raft_consensus.cc:2258] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:04.332998 2427 ts_tablet_manager.cc:1905] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:04.337378 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.317388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.16.21.1:40905)
0806 10:52:04.317553 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.317559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.328870 (+ 11311us) server_negotiation.cc:407] Connection header received
0806 10:52:04.329067 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.329075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.329195 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.329322 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:04.330050 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.331293 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.333251 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.333930 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.334030 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.335575 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:04.335586 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:04.335603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:04.336362 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:04.336423 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:04.336447 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:04.336552 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.336836 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.336952 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:52:04.338754 2111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:04.340554 2427 ts_tablet_manager.cc:1918] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:04.340756 2427 log.cc:1196] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/8f64a41c64c540d68241916c40902e85
I20260806 10:52:04.341290 2427 ts_tablet_manager.cc:1939] T 8f64a41c64c540d68241916c40902e85 P d3d50f7a21224df69bc5ee8407e753ba: Deleting consensus metadata
I20260806 10:52:04.342700 1911 catalog_manager.cc:4841] TS d3d50f7a21224df69bc5ee8407e753ba (127.16.21.3:40415): tablet 8f64a41c64c540d68241916c40902e85 (table test-table [id=87d7719c9524428d8e1630b044d95446]) successfully deleted
I20260806 10:52:04.429333 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1986
W20260806 10:52:04.457645 1912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.21.1:40905 (error 108)
I20260806 10:52:04.458381 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2120
I20260806 10:52:04.459944 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.459375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41182)
0806 10:52:04.459618 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.459759 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260806 10:52:04.460306 1912 catalog_manager.cc:4584] TS 9f9c33ee75fc4cd1919b5fa6a880107c (127.16.21.1:40905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f64a41c64c540d68241916c40902e85: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
I20260806 10:52:04.466224 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.459317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.16.21.2:38043)
0806 10:52:04.459503 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.459510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.459533 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:52:04.459757 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.459763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.459874 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.459998 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:04.460550 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.461345 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.463142 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.463536 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.463623 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.464894 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:04.464902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:04.464916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:04.465474 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:04.465521 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:04.465540 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:04.465625 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.465786 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.465877 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:52:04.466944 2245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:04.514181 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.513648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41184)
0806 10:52:04.513889 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.513990 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 10:52:04.515359 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.514909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41190)
0806 10:52:04.515084 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.515175 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 10:52:04.566032 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2120
I20260806 10:52:04.576086 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.575489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41200)
0806 10:52:04.575741 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.575860 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 10:52:04.577421 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.576955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41210)
0806 10:52:04.577156 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.577253 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 10:52:04.587881 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2253
I20260806 10:52:04.595818 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.588746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.16.21.3:40415)
0806 10:52:04.588938 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:04.588943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:04.589034 (+ 91us) server_negotiation.cc:407] Connection header received
0806 10:52:04.589237 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:04.589241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:04.589322 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:04.589401 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:04.589964 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.590761 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.592697 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:04.593107 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:04.593198 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:04.594374 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:04.594382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:04.594396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:04.595007 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:04.595055 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:04.595075 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:04.595162 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:04.595357 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 10:52:04.595433 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 10:52:04.596705 2379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:04.657922 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.657379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41212)
0806 10:52:04.657610 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.657719 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 10:52:04.659036 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.658597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41224)
0806 10:52:04.658801 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.658900 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 10:52:04.719122 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2253
I20260806 10:52:04.741825 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 1896
I20260806 10:52:04.742915 1957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:04.853065 2397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:04.852434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40905 (local address 127.0.0.1:41240)
0806 10:52:04.852741 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:04.852875 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260806 10:52:04.853048 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 1896
2026-08-06T10:52:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2838 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:04 dist-test-slave-0b58 krb5kdc[2436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:04 dist-test-slave-0b58 krb5kdc[2436](info): set up 2 sockets
Aug 06 10:52:04 dist-test-slave-0b58 krb5kdc[2436](info): commencing operation
krb5kdc: starting...
W20260806 10:52:05.968492 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:06 dist-test-slave-0b58 krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:06.285279 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:43081
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:06.636407 2450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:06.636691 2450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:06.636781 2450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:06.643807 2450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:06.643906 2450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:06.643967 2450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:06.644025 2450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:06.644069 2450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:06.653448 2450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:43081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:06.654624 2450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:06.655838 2450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:06.667486 2456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:06.667734 2455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:06.667950 2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:06.668398 2450 server_base.cc:1034] running on GCE node
I20260806 10:52:06.669152 2450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:06.670436 2450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:06.671664 2450 hybrid_clock.cc:648] HybridClock initialized: now 1786013526671628 us; error 45 us; skew 500 ppm
I20260806 10:52:06.672006 2450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:06 dist-test-slave-0b58 krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:06.675217 2450 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:06.676656 2450 webserver.cc:460] Webserver started at http://127.16.21.62:40447/ using document root <none> and password file <none>
I20260806 10:52:06.677526 2450 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:06.677682 2450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:06.678010 2450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:06.680969 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "eeb2d7a70fc54302a94d603de63a2b4a"
format_stamp: "Formatted at 2026-08-06 10:52:06 on dist-test-slave-0b58"
server_key: "7053bb4bf7e4182128a666ff0a0df230"
server_key_iv: "4920057ce18ac364f48774dcfa484bc2"
server_key_version: "encryptionkey@0"
I20260806 10:52:06.681735 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "eeb2d7a70fc54302a94d603de63a2b4a"
format_stamp: "Formatted at 2026-08-06 10:52:06 on dist-test-slave-0b58"
server_key: "7053bb4bf7e4182128a666ff0a0df230"
server_key_iv: "4920057ce18ac364f48774dcfa484bc2"
server_key_version: "encryptionkey@0"
I20260806 10:52:06.686887 2450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 10:52:06.690181 2465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:06.691349 2450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 10:52:06.691525 2450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "eeb2d7a70fc54302a94d603de63a2b4a"
format_stamp: "Formatted at 2026-08-06 10:52:06 on dist-test-slave-0b58"
server_key: "7053bb4bf7e4182128a666ff0a0df230"
server_key_iv: "4920057ce18ac364f48774dcfa484bc2"
server_key_version: "encryptionkey@0"
I20260806 10:52:06.691723 2450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:06.727487 2450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:06.731541 2450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:06.731827 2450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:06.750811 2450 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:43081
I20260806 10:52:06.750905 2517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:43081 every 8 connection(s)
I20260806 10:52:06.752607 2450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:06.754549 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2450
I20260806 10:52:06.754747 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:06.755172 16468 external_mini_cluster.cc:1428] Setting key 5a799161ddce320b028c4cd52027d81a
I20260806 10:52:06.756831 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:52:06 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:06.770987 2518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a: Bootstrap starting.
I20260806 10:52:06.775214 2518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:06.776528 2518 log.cc:824] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:06.779788 2518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a: No bootstrap required, opened a new log
I20260806 10:52:06.780030 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:06.757033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.16.21.62:43081)
0806 10:52:06.757853 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:06.757863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:06.757903 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:52:06.759522 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:06.759541 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:06.759999 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:06.760445 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:06.762413 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:06.763773 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:06.765668 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:06.766324 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:06.766448 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:06.771438 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:06.771450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:06.771469 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:06.771519 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:06.775567 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:06.776345 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:06.776355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:06.776364 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:06.776579 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:06.776996 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:06.777002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:06.777009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:06.777598 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:06.777926 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:06.778322 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 10:52:06.778744 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":378,"threads_started":1}
I20260806 10:52:06.787919 2518 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43081 } }
I20260806 10:52:06.788319 2518 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:06.788415 2518 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb2d7a70fc54302a94d603de63a2b4a, State: Initialized, Role: FOLLOWER
I20260806 10:52:06.788870 2518 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43081 } }
I20260806 10:52:06.789069 2518 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:06.789175 2518 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:06.789337 2518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:06.791611 2518 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43081 } }
I20260806 10:52:06.792083 2518 leader_election.cc:302] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeb2d7a70fc54302a94d603de63a2b4a; no voters:
I20260806 10:52:06.792704 2518 leader_election.cc:288] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:06.792852 2523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:06.794656 2523 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [term 1 LEADER]: Becoming Leader. State: Replica: eeb2d7a70fc54302a94d603de63a2b4a, State: Running, Role: LEADER
I20260806 10:52:06.795177 2523 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43081 } }
I20260806 10:52:06.795310 2518 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:06.798069 2525 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb2d7a70fc54302a94d603de63a2b4a. Latest consensus state: current_term: 1 leader_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43081 } } }
I20260806 10:52:06.798578 2525 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:06.799017 2524 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb2d7a70fc54302a94d603de63a2b4a" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 43081 } } }
I20260806 10:52:06.799782 2524 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:06.804104 2532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:06.809274 2532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:06.817059 2532 catalog_manager.cc:1347] Generated new cluster ID: 83e781fbe9a3433f9b184bd4fb9d6b6b
I20260806 10:52:06.817166 2532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:06.843873 2532 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:06.845759 2532 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:06.856388 2532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a: Generated new TSK 0
I20260806 10:52:06.857400 2532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:52:06.940133 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43081
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:07.281396 2544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:07.281667 2544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:07.281838 2544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:07.289147 2544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:07.289292 2544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:07.289484 2544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:52:07.299784 2544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:07.301118 2544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:07.302417 2544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:07.315991 2549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:07.315917 2553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:07.315953 2551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:07.316702 2544 server_base.cc:1034] running on GCE node
I20260806 10:52:07.317297 2544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:07.318210 2544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:07.319423 2544 hybrid_clock.cc:648] HybridClock initialized: now 1786013527319367 us; error 51 us; skew 500 ppm
I20260806 10:52:07.319729 2544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:07 dist-test-slave-0b58 krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013527, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:07.323138 2544 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:52:07.324744 2544 webserver.cc:460] Webserver started at http://127.16.21.1:39783/ using document root <none> and password file <none>
I20260806 10:52:07.325421 2544 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:07.325577 2544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:07.325911 2544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:07.328820 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "635ca0d3fa88428ab898f79fd37470ea"
format_stamp: "Formatted at 2026-08-06 10:52:07 on dist-test-slave-0b58"
server_key: "a5f44aea4756573b362ad447db8bb01b"
server_key_iv: "c068a405865c6f6b497aa8878c7c4c83"
server_key_version: "encryptionkey@0"
I20260806 10:52:07.329619 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "635ca0d3fa88428ab898f79fd37470ea"
format_stamp: "Formatted at 2026-08-06 10:52:07 on dist-test-slave-0b58"
server_key: "a5f44aea4756573b362ad447db8bb01b"
server_key_iv: "c068a405865c6f6b497aa8878c7c4c83"
server_key_version: "encryptionkey@0"
I20260806 10:52:07.334796 2544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:52:07.338171 2560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:07.339370 2544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:52:07.339552 2544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "635ca0d3fa88428ab898f79fd37470ea"
format_stamp: "Formatted at 2026-08-06 10:52:07 on dist-test-slave-0b58"
server_key: "a5f44aea4756573b362ad447db8bb01b"
server_key_iv: "c068a405865c6f6b497aa8878c7c4c83"
server_key_version: "encryptionkey@0"
I20260806 10:52:07.339761 2544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:07.361512 2544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:07.365777 2544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:07.366074 2544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:07.367056 2544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:07.368402 2544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:07.368512 2544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:07.368666 2544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:07.368746 2544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:07.403182 2544 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:35703
I20260806 10:52:07.403232 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:35703 every 8 connection(s)
I20260806 10:52:07.404516 2544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:52:07.412114 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2544
I20260806 10:52:07.412352 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:52:07.412793 16468 external_mini_cluster.cc:1428] Setting key 8fde60c06d7c7d111c00fe6df1a19a31
Aug 06 10:52:07 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013527, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:07.425904 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:07.407033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:42527 (local address 127.16.21.62:43081)
0806 10:52:07.407519 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:07.407524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:07.408337 (+ 813us) server_negotiation.cc:407] Connection header received
0806 10:52:07.410177 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:07.410183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:07.410280 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:07.410407 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:07.411938 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.412822 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.414558 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.415131 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.415302 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:07.418963 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:07.418973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:07.418981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:07.419028 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:07.422586 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:07.423317 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:07.423324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:07.423330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:07.423489 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:07.423892 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:07.423900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:07.423906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:07.424333 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:07.424495 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:07.425183 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 10:52:07.425452 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":281,"threads_started":1}
I20260806 10:52:07.426513 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:07.407435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:43081 (local address 127.16.21.1:42527)
0806 10:52:07.408141 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:07.408182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:07.409905 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:07.410648 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:07.410655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:07.411186 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:07.411732 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:07.411748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.412975 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.412980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:07.414382 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:07.414394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.415380 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.415385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:07.415579 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:07.416362 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:07.416383 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:07.418730 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:07.422728 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:07.422735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:07.422746 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:07.423188 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:07.423600 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:07.423603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:07.423605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:07.423773 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:07.424464 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:07.424470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:07.424767 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 10:52:07.425004 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 10:52:07.425395 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260806 10:52:07.428120 2675 heartbeater.cc:347] Connected to a master server at 127.16.21.62:43081
I20260806 10:52:07.428483 2675 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:07.429193 2675 heartbeater.cc:511] Master 127.16.21.62:43081 requested a full tablet report, sending...
I20260806 10:52:07.431429 2482 ts_manager.cc:194] Registered new tserver with Master: 635ca0d3fa88428ab898f79fd37470ea (127.16.21.1:35703)
I20260806 10:52:07.432930 2482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:42527
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:52:07.486042 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43081
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:07.828976 2681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:07.829326 2681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:07.829511 2681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:07.836755 2681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:07.836908 2681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:07.837096 2681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:52:07.846906 2681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:07.848186 2681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:07.849658 2681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:07.862813 2687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:07.862972 2689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:07.862839 2686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:07.863547 2681 server_base.cc:1034] running on GCE node
I20260806 10:52:07.864136 2681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:07.865068 2681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:07.866314 2681 hybrid_clock.cc:648] HybridClock initialized: now 1786013527866242 us; error 90 us; skew 500 ppm
I20260806 10:52:07.866633 2681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:07 dist-test-slave-0b58 krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013527, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:07.869772 2681 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:52:07.871254 2681 webserver.cc:460] Webserver started at http://127.16.21.2:44915/ using document root <none> and password file <none>
I20260806 10:52:07.871941 2681 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:07.872116 2681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:07.872524 2681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:07.875357 2681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "665e6e0006ac49688ad2302496af3d60"
format_stamp: "Formatted at 2026-08-06 10:52:07 on dist-test-slave-0b58"
server_key: "b32df4a7fcd8d697a518d0b6d4cf56ee"
server_key_iv: "dc18b900d269e6a2cc07a165a67fc72a"
server_key_version: "encryptionkey@0"
I20260806 10:52:07.876135 2681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "665e6e0006ac49688ad2302496af3d60"
format_stamp: "Formatted at 2026-08-06 10:52:07 on dist-test-slave-0b58"
server_key: "b32df4a7fcd8d697a518d0b6d4cf56ee"
server_key_iv: "dc18b900d269e6a2cc07a165a67fc72a"
server_key_version: "encryptionkey@0"
I20260806 10:52:07.881330 2681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 10:52:07.884696 2696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:07.885943 2681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:52:07.886132 2681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "665e6e0006ac49688ad2302496af3d60"
format_stamp: "Formatted at 2026-08-06 10:52:07 on dist-test-slave-0b58"
server_key: "b32df4a7fcd8d697a518d0b6d4cf56ee"
server_key_iv: "dc18b900d269e6a2cc07a165a67fc72a"
server_key_version: "encryptionkey@0"
I20260806 10:52:07.886381 2681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:07.907109 2681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:07.911371 2681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:07.911682 2681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:07.912753 2681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:07.914091 2681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:07.914213 2681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:07.914382 2681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:07.914462 2681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:07.948886 2681 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:32925
I20260806 10:52:07.948923 2810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:32925 every 8 connection(s)
I20260806 10:52:07.950160 2681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:52:07.959335 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2681
I20260806 10:52:07.959535 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:52:07.960000 16468 external_mini_cluster.cc:1428] Setting key 9907de8dd6f2fcbd8f32fa9cfee57cc4
Aug 06 10:52:07 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013527, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:07.969928 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:07.952714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58533 (local address 127.16.21.62:43081)
0806 10:52:07.953176 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:07.953181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:07.954106 (+ 925us) server_negotiation.cc:407] Connection header received
0806 10:52:07.955930 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:07.955936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:07.956025 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:07.956157 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:07.957654 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.958397 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.960086 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.960581 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.960716 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:07.964221 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:07.964226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:07.964230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:07.964310 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:07.966853 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:07.967561 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:07.967568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:07.967571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:07.967705 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:07.968141 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:07.968148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:07.968151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:07.968470 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:07.968647 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:07.969331 (+ 684us) server_negotiation.cc:299] Negotiation successful
0806 10:52:07.969547 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260806 10:52:07.970845 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:07.953237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:43081 (local address 127.16.21.2:58533)
0806 10:52:07.953947 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:07.953989 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:07.955653 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:07.956426 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:07.956433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:07.956951 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:07.957470 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:07.957487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.958572 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.958576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:07.959940 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:07.959952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:07.960829 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:07.960837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:07.961110 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:07.961845 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:07.961875 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:07.963997 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:07.966998 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:07.967007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:07.967022 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:07.967432 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:07.967828 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:07.967834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:07.967837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:07.968018 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:07.968604 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:07.968614 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:07.968948 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 10:52:07.969217 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 10:52:07.969715 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":274,"threads_started":1}
I20260806 10:52:07.972440 2811 heartbeater.cc:347] Connected to a master server at 127.16.21.62:43081
I20260806 10:52:07.972743 2811 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:07.973431 2811 heartbeater.cc:511] Master 127.16.21.62:43081 requested a full tablet report, sending...
I20260806 10:52:07.975023 2482 ts_manager.cc:194] Registered new tserver with Master: 665e6e0006ac49688ad2302496af3d60 (127.16.21.2:32925)
I20260806 10:52:07.976276 2482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:58533
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:52:08.032426 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43081
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:08.369710 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:08.369961 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:08.370131 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:08.377218 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:08.377310 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:08.377516 2817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:52:08.387115 2817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:08.388350 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:08.389715 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:08.403414 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:08.404052 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:08.404624 2817 server_base.cc:1034] running on GCE node
W20260806 10:52:08.404841 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:08.405543 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:08.406539 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:08.407789 2817 hybrid_clock.cc:648] HybridClock initialized: now 1786013528407748 us; error 32 us; skew 500 ppm
I20260806 10:52:08.408118 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:08 dist-test-slave-0b58 krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013528, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:08.411193 2817 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:52:08.412752 2817 webserver.cc:460] Webserver started at http://127.16.21.3:43157/ using document root <none> and password file <none>
I20260806 10:52:08.413424 2817 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:08.413581 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:08.413909 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:08.416816 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "d6250905dbc64285a009c78450f8981c"
format_stamp: "Formatted at 2026-08-06 10:52:08 on dist-test-slave-0b58"
server_key: "320d2004335cc65b6b63fed1061f258d"
server_key_iv: "d9aaeee7cf772b938f9bd39d8d2eda27"
server_key_version: "encryptionkey@0"
I20260806 10:52:08.417574 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "d6250905dbc64285a009c78450f8981c"
format_stamp: "Formatted at 2026-08-06 10:52:08 on dist-test-slave-0b58"
server_key: "320d2004335cc65b6b63fed1061f258d"
server_key_iv: "d9aaeee7cf772b938f9bd39d8d2eda27"
server_key_version: "encryptionkey@0"
I20260806 10:52:08.422780 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 10:52:08.426326 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:08.427520 2817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:52:08.427719 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "d6250905dbc64285a009c78450f8981c"
format_stamp: "Formatted at 2026-08-06 10:52:08 on dist-test-slave-0b58"
server_key: "320d2004335cc65b6b63fed1061f258d"
server_key_iv: "d9aaeee7cf772b938f9bd39d8d2eda27"
server_key_version: "encryptionkey@0"
I20260806 10:52:08.427937 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:08.436501 2675 heartbeater.cc:503] Master 127.16.21.62:43081 was elected leader, sending a full tablet report...
I20260806 10:52:08.450850 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:08.454684 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:08.454990 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:08.455936 2817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:08.457273 2817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:08.457383 2817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:08.457542 2817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:08.457621 2817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:08.492651 2817 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:33071
I20260806 10:52:08.492739 2944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:33071 every 8 connection(s)
I20260806 10:52:08.493994 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:52:08.502985 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2817
I20260806 10:52:08.503167 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:52:08.503607 16468 external_mini_cluster.cc:1428] Setting key 18270a2e1976ec714149d4fb2c350fa7
Aug 06 10:52:08 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013528, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:08.514196 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.496722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:60927 (local address 127.16.21.62:43081)
0806 10:52:08.497201 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.497205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.498068 (+ 863us) server_negotiation.cc:407] Connection header received
0806 10:52:08.500076 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.500082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.500175 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.500335 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:08.501832 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.502684 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.504430 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.505000 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.505149 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.508754 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:08.508759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:08.508762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:08.508795 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:08.511559 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.512226 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.512232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.512235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.512402 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.512811 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.512814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.512816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.513075 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:08.513187 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.513769 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.513932 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":250,"threads_started":1}
I20260806 10:52:08.515090 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.497132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:43081 (local address 127.16.21.3:60927)
0806 10:52:08.497911 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:08.497952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:08.499807 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:08.500579 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:08.500587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:08.501088 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:08.501636 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.501658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.502855 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.502859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.504244 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.504304 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.505244 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.505249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.505542 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.506286 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:08.506316 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:08.508571 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:08.511678 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:08.511683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:08.511693 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:08.512086 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:08.512513 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:08.512516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:08.512518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:08.512688 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:08.513180 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:08.513185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:08.513464 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 10:52:08.513666 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 10:52:08.514013 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":309,"threads_started":1}
I20260806 10:52:08.516436 2945 heartbeater.cc:347] Connected to a master server at 127.16.21.62:43081
I20260806 10:52:08.516759 2945 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:08.517612 2945 heartbeater.cc:511] Master 127.16.21.62:43081 requested a full tablet report, sending...
I20260806 10:52:08.519263 2482 ts_manager.cc:194] Registered new tserver with Master: d6250905dbc64285a009c78450f8981c (127.16.21.3:33071)
I20260806 10:52:08.520025 2482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:60927
I20260806 10:52:08.531138 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:52:08.548318 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.535501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.16.21.62:43081)
0806 10:52:08.535741 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.535746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.535996 (+ 250us) server_negotiation.cc:407] Connection header received
0806 10:52:08.536115 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.536119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.536181 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.536368 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:08.537791 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.538512 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.539797 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.540192 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.540364 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.543045 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:08.543050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:08.543053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:08.543082 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:08.545703 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.546450 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.546453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.546456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.546565 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.546959 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.546963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.546965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.547247 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:08.547443 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.547779 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.547935 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:52:08.552428 2482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:52:08.559216 2482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:52:08.588744 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.571325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.16.21.3:33071)
0806 10:52:08.578257 (+ 6932us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.578271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.578990 (+ 719us) server_negotiation.cc:407] Connection header received
0806 10:52:08.579119 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.579129 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.579402 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.581296 (+ 1894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:08.581634 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.583286 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.586210 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.587547 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.587684 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.587937 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.588195 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.588393 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6684,"thread_start_us":6503,"threads_started":1}
I20260806 10:52:08.588828 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.571095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33071 (local address 127.0.0.1:51540)
0806 10:52:08.578641 (+ 7546us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:08.578676 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:08.578866 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:08.580238 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:08.580245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:08.580299 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:08.580838 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.580861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.583449 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.583454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.585952 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.585971 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.587743 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.587749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.587943 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.587988 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 10:52:08.588098 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 10:52:08.588318 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7305,"mutex_wait_us":18,"thread_start_us":8046,"threads_started":1}
I20260806 10:52:08.590153 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.571364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:35703 (local address 127.0.0.1:41146)
0806 10:52:08.579662 (+ 8298us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:08.579685 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:08.579827 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:08.581653 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:08.581659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:08.581687 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:08.582147 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.582160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.585130 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.585137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.587339 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.587354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.589368 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.589375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.589571 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.589604 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:52:08.589694 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 10:52:08.589815 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8146,"thread_start_us":6913,"threads_started":1}
I20260806 10:52:08.590952 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.572031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.16.21.1:35703)
0806 10:52:08.580850 (+ 8819us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.580859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.580890 (+ 31us) server_negotiation.cc:407] Connection header received
0806 10:52:08.580989 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.580996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.581200 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.581418 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:08.583420 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.584987 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.587524 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.588992 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.589103 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.589343 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.590432 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.590605 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8590,"thread_start_us":8988,"threads_started":1}
I20260806 10:52:08.593253 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.580093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:32925 (local address 127.0.0.1:55102)
0806 10:52:08.585002 (+ 4909us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:08.585014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:08.585152 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:08.585751 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:08.585756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:08.585782 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:08.586552 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.586565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.588400 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.588405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.590596 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.590610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.592431 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.592437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.592599 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.592628 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 10:52:08.592741 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 10:52:08.592854 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4784,"mutex_wait_us":82,"thread_start_us":4605,"threads_started":1}
I20260806 10:52:08.593264 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.579412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.16.21.2:32925)
0806 10:52:08.584662 (+ 5250us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.584673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.585165 (+ 492us) server_negotiation.cc:407] Connection header received
0806 10:52:08.585307 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.585316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.585528 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.585712 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:08.586746 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.588220 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.590804 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.592194 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.592384 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.592572 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.592785 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.592939 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5083,"thread_start_us":4965,"threads_started":1}
I20260806 10:52:08.593329 2880 tablet_service.cc:1467] Processing CreateTablet for tablet 202d41f9327a4b5e8addfd23fe818628 (DEFAULT_TABLE table=test-table [id=80446630c62740c7b37d21c8ada44ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:08.594899 2880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202d41f9327a4b5e8addfd23fe818628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:08.594605 2609 tablet_service.cc:1467] Processing CreateTablet for tablet 202d41f9327a4b5e8addfd23fe818628 (DEFAULT_TABLE table=test-table [id=80446630c62740c7b37d21c8ada44ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:08.596112 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202d41f9327a4b5e8addfd23fe818628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:08.596928 2745 tablet_service.cc:1467] Processing CreateTablet for tablet 202d41f9327a4b5e8addfd23fe818628 (DEFAULT_TABLE table=test-table [id=80446630c62740c7b37d21c8ada44ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:08.598022 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202d41f9327a4b5e8addfd23fe818628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:08.610231 2964 tablet_bootstrap.cc:492] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Bootstrap starting.
I20260806 10:52:08.610888 2965 tablet_bootstrap.cc:492] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Bootstrap starting.
I20260806 10:52:08.611670 2966 tablet_bootstrap.cc:492] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Bootstrap starting.
I20260806 10:52:08.614297 2964 tablet_bootstrap.cc:654] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:08.614809 2965 tablet_bootstrap.cc:654] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:08.615051 2966 tablet_bootstrap.cc:654] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:08.615568 2964 log.cc:824] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:08.615890 2965 log.cc:824] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:08.616222 2966 log.cc:824] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:08.619503 2966 tablet_bootstrap.cc:492] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: No bootstrap required, opened a new log
I20260806 10:52:08.619793 2966 ts_tablet_manager.cc:1397] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 10:52:08.620469 2965 tablet_bootstrap.cc:492] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: No bootstrap required, opened a new log
I20260806 10:52:08.620479 2964 tablet_bootstrap.cc:492] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: No bootstrap required, opened a new log
I20260806 10:52:08.620733 2964 ts_tablet_manager.cc:1397] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260806 10:52:08.620733 2965 ts_tablet_manager.cc:1397] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260806 10:52:08.626768 2966 raft_consensus.cc:348] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.627135 2966 raft_consensus.cc:374] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:08.627264 2966 raft_consensus.cc:729] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6250905dbc64285a009c78450f8981c, State: Initialized, Role: FOLLOWER
I20260806 10:52:08.627892 2966 consensus_queue.cc:260] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.630496 2945 heartbeater.cc:503] Master 127.16.21.62:43081 was elected leader, sending a full tablet report...
I20260806 10:52:08.631214 2966 ts_tablet_manager.cc:1428] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Time spent starting tablet: real 0.011s user 0.000s sys 0.011s
I20260806 10:52:08.631502 2964 raft_consensus.cc:348] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.631508 2965 raft_consensus.cc:348] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.631924 2965 raft_consensus.cc:374] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:08.632006 2964 raft_consensus.cc:374] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:08.632074 2965 raft_consensus.cc:729] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635ca0d3fa88428ab898f79fd37470ea, State: Initialized, Role: FOLLOWER
I20260806 10:52:08.632123 2964 raft_consensus.cc:729] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665e6e0006ac49688ad2302496af3d60, State: Initialized, Role: FOLLOWER
I20260806 10:52:08.632754 2965 consensus_queue.cc:260] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.632848 2964 consensus_queue.cc:260] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.634802 2965 ts_tablet_manager.cc:1428] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 10:52:08.636229 2811 heartbeater.cc:503] Master 127.16.21.62:43081 was elected leader, sending a full tablet report...
I20260806 10:52:08.637310 2964 ts_tablet_manager.cc:1428] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 10:52:08.655138 2971 raft_consensus.cc:482] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:52:08.655382 2971 raft_consensus.cc:504] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
W20260806 10:52:08.657781 2676 tablet.cc:2369] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:08.658666 2971 leader_election.cc:288] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6250905dbc64285a009c78450f8981c (127.16.21.3:33071), 665e6e0006ac49688ad2302496af3d60 (127.16.21.2:32925)
I20260806 10:52:08.665165 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.657513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:37197 (local address 127.16.21.3:33071)
0806 10:52:08.657704 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.657713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.659741 (+ 2028us) server_negotiation.cc:407] Connection header received
0806 10:52:08.659916 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.659924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.659998 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.660101 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:08.660801 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.661570 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.663389 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.664147 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.664244 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.664415 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.664761 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.664907 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 10:52:08.665223 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.657618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:54875 (local address 127.16.21.2:32925)
0806 10:52:08.657842 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.657848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.659557 (+ 1709us) server_negotiation.cc:407] Connection header received
0806 10:52:08.659745 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.659752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.659834 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.659922 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:08.660636 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.661406 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.663395 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.664178 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.664377 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.664471 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.664849 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.664989 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:52:08.665246 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.657411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33071 (local address 127.16.21.1:37197)
0806 10:52:08.659605 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:08.659626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:08.659770 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:08.660130 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:08.660134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:08.660171 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:08.660637 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.660649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.661716 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.661719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.663232 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.663244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.664414 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.664422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.664584 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.664626 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 10:52:08.664700 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 10:52:08.664829 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2077,"mutex_wait_us":35,"spinlock_wait_cycles":2432,"thread_start_us":1741,"threads_started":1}
I20260806 10:52:08.665712 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.657354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:32925 (local address 127.16.21.1:54875)
0806 10:52:08.659464 (+ 2110us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:08.659486 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:08.659602 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:08.659953 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:08.659958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:08.659982 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:08.660471 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.660482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.661547 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.661552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.663220 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:08.663232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.664466 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.664472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:08.664655 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.664704 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 10:52:08.664801 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 10:52:08.665317 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"mutex_wait_us":121,"spinlock_wait_cycles":24960,"thread_start_us":1706,"threads_started":1}
I20260806 10:52:08.666534 2900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d41f9327a4b5e8addfd23fe818628" candidate_uuid: "635ca0d3fa88428ab898f79fd37470ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6250905dbc64285a009c78450f8981c" is_pre_election: true
I20260806 10:52:08.666813 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d41f9327a4b5e8addfd23fe818628" candidate_uuid: "635ca0d3fa88428ab898f79fd37470ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665e6e0006ac49688ad2302496af3d60" is_pre_election: true
I20260806 10:52:08.667040 2900 raft_consensus.cc:2454] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635ca0d3fa88428ab898f79fd37470ea in term 0.
I20260806 10:52:08.667318 2765 raft_consensus.cc:2454] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635ca0d3fa88428ab898f79fd37470ea in term 0.
I20260806 10:52:08.667812 2562 leader_election.cc:302] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635ca0d3fa88428ab898f79fd37470ea, d6250905dbc64285a009c78450f8981c; no voters:
I20260806 10:52:08.668311 2971 raft_consensus.cc:2781] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:52:08.668478 2971 raft_consensus.cc:482] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:52:08.668599 2971 raft_consensus.cc:3037] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:08.670821 2971 raft_consensus.cc:504] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.671409 2971 leader_election.cc:288] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [CANDIDATE]: Term 1 election: Requested vote from peers d6250905dbc64285a009c78450f8981c (127.16.21.3:33071), 665e6e0006ac49688ad2302496af3d60 (127.16.21.2:32925)
I20260806 10:52:08.671880 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d41f9327a4b5e8addfd23fe818628" candidate_uuid: "635ca0d3fa88428ab898f79fd37470ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665e6e0006ac49688ad2302496af3d60"
I20260806 10:52:08.671880 2900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d41f9327a4b5e8addfd23fe818628" candidate_uuid: "635ca0d3fa88428ab898f79fd37470ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6250905dbc64285a009c78450f8981c"
I20260806 10:52:08.672089 2900 raft_consensus.cc:3037] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:08.672089 2765 raft_consensus.cc:3037] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:08.675341 2765 raft_consensus.cc:2454] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635ca0d3fa88428ab898f79fd37470ea in term 1.
I20260806 10:52:08.675341 2900 raft_consensus.cc:2454] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635ca0d3fa88428ab898f79fd37470ea in term 1.
I20260806 10:52:08.675768 2563 leader_election.cc:302] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635ca0d3fa88428ab898f79fd37470ea, 665e6e0006ac49688ad2302496af3d60; no voters:
I20260806 10:52:08.676120 2971 raft_consensus.cc:2781] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:08.676606 2971 raft_consensus.cc:686] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 1 LEADER]: Becoming Leader. State: Replica: 635ca0d3fa88428ab898f79fd37470ea, State: Running, Role: LEADER
I20260806 10:52:08.677089 2971 consensus_queue.cc:237] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } }
I20260806 10:52:08.681751 2481 catalog_manager.cc:5495] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 635ca0d3fa88428ab898f79fd37470ea (127.16.21.1). New cstate: current_term: 1 leader_uuid: "635ca0d3fa88428ab898f79fd37470ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "635ca0d3fa88428ab898f79fd37470ea" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 35703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 } health_report { overall_health: UNKNOWN } } }
W20260806 10:52:08.702251 2812 tablet.cc:2369] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:08.714196 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.709026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.16.21.1:35703)
0806 10:52:08.709194 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.709198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.709217 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:52:08.709367 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.709371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.709452 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.709569 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:52:08.709913 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.710622 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.711931 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.712372 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.712526 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.713114 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:52:08.713591 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:52:08.713716 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.713868 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.713942 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 10:52:08.722069 2765 raft_consensus.cc:1261] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 1 FOLLOWER]: Refusing update from remote peer 635ca0d3fa88428ab898f79fd37470ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:08.722069 2900 raft_consensus.cc:1261] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 1 FOLLOWER]: Refusing update from remote peer 635ca0d3fa88428ab898f79fd37470ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:08.723048 2975 consensus_queue.cc:1035] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "665e6e0006ac49688ad2302496af3d60" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:08.723407 2971 consensus_queue.cc:1035] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6250905dbc64285a009c78450f8981c" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:08.749365 2482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:08.749666 2482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:08.753821 2482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eeb2d7a70fc54302a94d603de63a2b4a: Sending DeleteTablet for 3 replicas of tablet 202d41f9327a4b5e8addfd23fe818628
I20260806 10:52:08.755116 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet 202d41f9327a4b5e8addfd23fe818628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:08 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:55102
I20260806 10:52:08.755170 2880 tablet_service.cc:1514] Processing DeleteTablet for tablet 202d41f9327a4b5e8addfd23fe818628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:08 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:51540
I20260806 10:52:08.756240 2609 tablet_service.cc:1514] Processing DeleteTablet for tablet 202d41f9327a4b5e8addfd23fe818628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:08 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:41146
I20260806 10:52:08.762589 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2544
I20260806 10:52:08.768788 2990 tablet_replica.cc:330] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: stopping tablet replica
I20260806 10:52:08.769291 2990 raft_consensus.cc:2229] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:52:08.770000 2990 raft_consensus.cc:2258] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:08.772367 2990 ts_tablet_manager.cc:1905] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:08.788601 2990 ts_tablet_manager.cc:1918] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:08.788856 2990 log.cc:1196] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/202d41f9327a4b5e8addfd23fe818628
I20260806 10:52:08.789412 2989 tablet_replica.cc:330] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: stopping tablet replica
I20260806 10:52:08.789528 2990 ts_tablet_manager.cc:1939] T 202d41f9327a4b5e8addfd23fe818628 P 635ca0d3fa88428ab898f79fd37470ea: Deleting consensus metadata
I20260806 10:52:08.791356 2989 raft_consensus.cc:2229] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:08.791843 2468 catalog_manager.cc:4841] TS 635ca0d3fa88428ab898f79fd37470ea (127.16.21.1:35703): tablet 202d41f9327a4b5e8addfd23fe818628 (table test-table [id=80446630c62740c7b37d21c8ada44ac1]) successfully deleted
I20260806 10:52:08.792194 2989 raft_consensus.cc:2258] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:08.792951 2988 tablet_replica.cc:330] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: stopping tablet replica
I20260806 10:52:08.793326 2988 raft_consensus.cc:2229] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:08.793768 2988 raft_consensus.cc:2258] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:08.796096 2989 ts_tablet_manager.cc:1905] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:08.796096 2988 ts_tablet_manager.cc:1905] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 10:52:08 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:52:08.803483 2988 ts_tablet_manager.cc:1918] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:08.803675 2988 log.cc:1196] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/202d41f9327a4b5e8addfd23fe818628
I20260806 10:52:08.804147 2988 ts_tablet_manager.cc:1939] T 202d41f9327a4b5e8addfd23fe818628 P 665e6e0006ac49688ad2302496af3d60: Deleting consensus metadata
I20260806 10:52:08.804793 2989 ts_tablet_manager.cc:1918] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:08.804950 2989 log.cc:1196] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/202d41f9327a4b5e8addfd23fe818628
I20260806 10:52:08.805469 2989 ts_tablet_manager.cc:1939] T 202d41f9327a4b5e8addfd23fe818628 P d6250905dbc64285a009c78450f8981c: Deleting consensus metadata
I20260806 10:52:08.806021 2468 catalog_manager.cc:4841] TS 665e6e0006ac49688ad2302496af3d60 (127.16.21.2:32925): tablet 202d41f9327a4b5e8addfd23fe818628 (table test-table [id=80446630c62740c7b37d21c8ada44ac1]) successfully deleted
I20260806 10:52:08.807154 2467 catalog_manager.cc:4841] TS d6250905dbc64285a009c78450f8981c (127.16.21.3:33071): tablet 202d41f9327a4b5e8addfd23fe818628 (table test-table [id=80446630c62740c7b37d21c8ada44ac1]) successfully deleted
I20260806 10:52:08.809902 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.763983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.16.21.1:35703)
0806 10:52:08.764193 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.764200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.787570 (+ 23370us) server_negotiation.cc:407] Connection header received
0806 10:52:08.788007 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.788013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.788105 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.788182 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:08.792411 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.793654 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.795845 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.796555 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.796671 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.802567 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:08.802583 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:08.802601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:08.802644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:08.806810 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.807642 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.807653 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.807656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.807834 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.808281 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.808287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.808290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.808648 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:08.808964 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.809311 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.809551 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:52:08.810611 2670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:08.910354 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2544
I20260806 10:52:08.929554 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2681
Aug 06 10:52:08 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:52:08.944614 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:08.930351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.16.21.2:32925)
0806 10:52:08.930559 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:08.930565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:08.930586 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:52:08.930741 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:08.930746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:08.930819 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:08.930927 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:08.932336 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.933071 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.934461 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:08.934861 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:08.934997 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:08.939175 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:08.939205 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:08.939219 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:08.939259 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:08.942079 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.942721 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.942728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.942730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.942864 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:08.943168 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:08.943171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:08.943173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:08.943540 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:08.943751 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:08.943994 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 10:52:08.944144 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:52:08.945353 2806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:09.035130 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2681
I20260806 10:52:09.055512 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2817
Aug 06 10:52:09 dist-test-slave-0b58 krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:52:09.070616 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:09.056362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.16.21.3:33071)
0806 10:52:09.056555 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:09.056561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:09.056586 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:52:09.056666 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:09.056670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:09.056736 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:09.056849 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:09.058103 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:09.058848 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:09.060139 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:09.060666 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:09.060809 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:09.065215 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:09.065230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:09.065246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:09.065291 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:09.068109 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:09.068841 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:09.068848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:09.068850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:09.068976 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:09.069302 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:09.069306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:09.069308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:09.069638 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:09.069846 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:09.070071 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 10:52:09.070237 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:52:09.071239 2940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:09.159979 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2817
I20260806 10:52:09.180831 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2450
I20260806 10:52:09.181914 2512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:09.294600 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2450
2026-08-06T10:52:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4442 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T10:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:09Z Disabled control of system clock
I20260806 10:52:09.389506 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44447
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:52:09.716156 2999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:09.716436 2999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:09.716522 2999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:09.723290 2999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:09.723380 2999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:09.723418 2999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:09.723466 2999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:09.723502 2999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:09.732081 2999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:44447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:09.733314 2999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:09.734979 2999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:09.746726 3004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:09.746821 3007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:09.746874 3005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:09.747711 2999 server_base.cc:1034] running on GCE node
I20260806 10:52:09.748558 2999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:09.749881 2999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:09.751086 2999 hybrid_clock.cc:648] HybridClock initialized: now 1786013529751049 us; error 33 us; skew 500 ppm
I20260806 10:52:09.751446 2999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:09.753711 2999 webserver.cc:460] Webserver started at http://127.16.21.62:46143/ using document root <none> and password file <none>
I20260806 10:52:09.754400 2999 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:09.754521 2999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:09.754864 2999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:09.757766 2999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3"
format_stamp: "Formatted at 2026-08-06 10:52:09 on dist-test-slave-0b58"
server_key: "f4a86beb2f370fc33a7c35a87e076d14"
server_key_iv: "f948dd8a473d3251da927af9b35e19d0"
server_key_version: "encryptionkey@0"
I20260806 10:52:09.758598 2999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3"
format_stamp: "Formatted at 2026-08-06 10:52:09 on dist-test-slave-0b58"
server_key: "f4a86beb2f370fc33a7c35a87e076d14"
server_key_iv: "f948dd8a473d3251da927af9b35e19d0"
server_key_version: "encryptionkey@0"
I20260806 10:52:09.764060 2999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:52:09.767314 3013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:09.768640 2999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:09.768832 2999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3"
format_stamp: "Formatted at 2026-08-06 10:52:09 on dist-test-slave-0b58"
server_key: "f4a86beb2f370fc33a7c35a87e076d14"
server_key_iv: "f948dd8a473d3251da927af9b35e19d0"
server_key_version: "encryptionkey@0"
I20260806 10:52:09.769059 2999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:09.783507 2999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:09.784302 2999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:09.784591 2999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:09.803601 2999 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:44447
I20260806 10:52:09.803632 3064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:44447 every 8 connection(s)
I20260806 10:52:09.804980 2999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:09.808616 3065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:09.815270 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 2999
I20260806 10:52:09.815462 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:09.815893 16468 external_mini_cluster.cc:1428] Setting key de8241c1051d25e910561f82542d473e
I20260806 10:52:09.819089 3065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3: Bootstrap starting.
I20260806 10:52:09.822476 3065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:09.823490 3065 log.cc:824] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:09.826501 3065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3: No bootstrap required, opened a new log
I20260806 10:52:09.829600 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:09.817920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.16.21.62:44447)
0806 10:52:09.818870 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:09.818881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:09.818917 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:52:09.820755 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:09.820780 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:09.821189 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:09.821477 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:09.822122 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:09.823486 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:09.825305 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:09.825952 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:09.826083 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:09.827347 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:09.827354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:09.827367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:09.827863 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:09.827944 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:09.827966 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:09.828063 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:09.828365 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 10:52:09.828697 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":339,"threads_started":1}
I20260806 10:52:09.834420 3065 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44447 } }
I20260806 10:52:09.834758 3065 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:09.834884 3065 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1c13c1fd6440459c8dd6e9b5f612e3, State: Initialized, Role: FOLLOWER
I20260806 10:52:09.835419 3065 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44447 } }
I20260806 10:52:09.835700 3065 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:09.835860 3065 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:09.836028 3065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:09.838595 3065 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44447 } }
I20260806 10:52:09.839128 3065 leader_election.cc:302] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa1c13c1fd6440459c8dd6e9b5f612e3; no voters:
I20260806 10:52:09.839708 3065 leader_election.cc:288] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:09.839874 3070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:09.841046 3070 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [term 1 LEADER]: Becoming Leader. State: Replica: aa1c13c1fd6440459c8dd6e9b5f612e3, State: Running, Role: LEADER
I20260806 10:52:09.841555 3070 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44447 } }
I20260806 10:52:09.841852 3065 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:09.844769 3072 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa1c13c1fd6440459c8dd6e9b5f612e3. Latest consensus state: current_term: 1 leader_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44447 } } }
I20260806 10:52:09.844830 3071 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1c13c1fd6440459c8dd6e9b5f612e3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44447 } } }
I20260806 10:52:09.845253 3071 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:09.845258 3072 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:09.846876 3079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:09.851766 3079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:09.860817 3079 catalog_manager.cc:1347] Generated new cluster ID: 92801663075b4817839498fe5ef88bc7
I20260806 10:52:09.860919 3079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:09.869982 3079 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:09.871206 3079 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:09.880134 3079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3: Generated new TSK 0
I20260806 10:52:09.880861 3079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:09.887677 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44447
--builtin_ntp_servers=127.16.21.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:52:10.232630 3090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:10.232949 3090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:10.233110 3090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:10.240899 3090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:10.241030 3090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:10.241215 3090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:52:10.250780 3090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:10.251992 3090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:10.253839 3090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:10.266470 3096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:10.266515 3098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:10.267165 3095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:10.268054 3090 server_base.cc:1034] running on GCE node
I20260806 10:52:10.268707 3090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:10.269603 3090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:10.270859 3090 hybrid_clock.cc:648] HybridClock initialized: now 1786013530270814 us; error 60 us; skew 500 ppm
I20260806 10:52:10.271178 3090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:10.273478 3090 webserver.cc:460] Webserver started at http://127.16.21.1:36991/ using document root <none> and password file <none>
I20260806 10:52:10.274127 3090 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:10.274264 3090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:10.274572 3090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:10.277473 3090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "f3871c452af64a33aaf63ca98cae7ffc"
format_stamp: "Formatted at 2026-08-06 10:52:10 on dist-test-slave-0b58"
server_key: "2b77ca01571f32fced9c6f9b0fc30a3a"
server_key_iv: "85fe3006c5bdc7b3b270b06dd723bbd9"
server_key_version: "encryptionkey@0"
I20260806 10:52:10.278293 3090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "f3871c452af64a33aaf63ca98cae7ffc"
format_stamp: "Formatted at 2026-08-06 10:52:10 on dist-test-slave-0b58"
server_key: "2b77ca01571f32fced9c6f9b0fc30a3a"
server_key_iv: "85fe3006c5bdc7b3b270b06dd723bbd9"
server_key_version: "encryptionkey@0"
I20260806 10:52:10.283342 3090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 10:52:10.286667 3104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:10.287889 3090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:52:10.288069 3090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "f3871c452af64a33aaf63ca98cae7ffc"
format_stamp: "Formatted at 2026-08-06 10:52:10 on dist-test-slave-0b58"
server_key: "2b77ca01571f32fced9c6f9b0fc30a3a"
server_key_iv: "85fe3006c5bdc7b3b270b06dd723bbd9"
server_key_version: "encryptionkey@0"
I20260806 10:52:10.288297 3090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:10.303609 3090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:10.304451 3090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:10.304726 3090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:10.305697 3090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:10.307196 3090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:10.307329 3090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:10.307473 3090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:10.307579 3090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:10.341401 3090 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:40875
I20260806 10:52:10.341492 3218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:40875 every 8 connection(s)
I20260806 10:52:10.342769 3090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:52:10.352667 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3090
I20260806 10:52:10.352833 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:52:10.353400 16468 external_mini_cluster.cc:1428] Setting key 015de02b7d3518d6c7b645b125e92010
I20260806 10:52:10.355496 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:10.345355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:40065 (local address 127.16.21.62:44447)
0806 10:52:10.345771 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:10.345782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:10.347150 (+ 1368us) server_negotiation.cc:407] Connection header received
0806 10:52:10.348998 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:10.349002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:10.349065 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:10.349168 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:10.350141 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.350843 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.352513 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.353210 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.353344 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:10.354667 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:10.354677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:10.354680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:10.354696 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:10.354736 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:10.354748 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:10.354844 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:10.355084 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 10:52:10.355167 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":217,"threads_started":1}
I20260806 10:52:10.356134 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44447
--builtin_ntp_servers=127.16.21.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:52:10.356477 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:10.345934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44447 (local address 127.16.21.1:40065)
0806 10:52:10.346990 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:10.347034 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:10.348761 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:10.349369 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:10.349376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:10.349455 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:10.349994 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:10.350011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.350978 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.350982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:10.352371 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:10.352383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.353414 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.353419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:10.353659 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:10.354339 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:10.354342 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:10.354403 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:10.354493 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:10.354846 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:10.354860 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:10.354868 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:52:10.355069 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 10:52:10.355403 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":386,"threads_started":1}
I20260806 10:52:10.358148 3219 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44447
I20260806 10:52:10.358443 3219 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:10.359301 3219 heartbeater.cc:511] Master 127.16.21.62:44447 requested a full tablet report, sending...
I20260806 10:52:10.361311 3030 ts_manager.cc:194] Registered new tserver with Master: f3871c452af64a33aaf63ca98cae7ffc (127.16.21.1:40875)
I20260806 10:52:10.362733 3030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.1:40065
W20260806 10:52:10.687870 3223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:10.688167 3223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:10.688360 3223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:10.695664 3223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:10.695789 3223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:10.695973 3223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:52:10.704746 3223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:10.705961 3223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:10.707706 3223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:10.720543 3228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:10.720605 3229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:10.720582 3231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:10.721465 3223 server_base.cc:1034] running on GCE node
I20260806 10:52:10.722028 3223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:10.722949 3223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:10.724208 3223 hybrid_clock.cc:648] HybridClock initialized: now 1786013530724149 us; error 45 us; skew 500 ppm
I20260806 10:52:10.724592 3223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:10.726873 3223 webserver.cc:460] Webserver started at http://127.16.21.2:32939/ using document root <none> and password file <none>
I20260806 10:52:10.727540 3223 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:10.727684 3223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:10.728008 3223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:10.731212 3223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "9d471cb334964b84b952f16f84b81359"
format_stamp: "Formatted at 2026-08-06 10:52:10 on dist-test-slave-0b58"
server_key: "37ba31a80c2347c6bdfcd89ec6d21a4c"
server_key_iv: "0c0bb5f43bc8e861e331b0aa5f075d19"
server_key_version: "encryptionkey@0"
I20260806 10:52:10.732028 3223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "9d471cb334964b84b952f16f84b81359"
format_stamp: "Formatted at 2026-08-06 10:52:10 on dist-test-slave-0b58"
server_key: "37ba31a80c2347c6bdfcd89ec6d21a4c"
server_key_iv: "0c0bb5f43bc8e861e331b0aa5f075d19"
server_key_version: "encryptionkey@0"
I20260806 10:52:10.737093 3223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 10:52:10.740617 3237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:10.741844 3223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:52:10.742079 3223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "9d471cb334964b84b952f16f84b81359"
format_stamp: "Formatted at 2026-08-06 10:52:10 on dist-test-slave-0b58"
server_key: "37ba31a80c2347c6bdfcd89ec6d21a4c"
server_key_iv: "0c0bb5f43bc8e861e331b0aa5f075d19"
server_key_version: "encryptionkey@0"
I20260806 10:52:10.742300 3223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:10.756641 3223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:10.757351 3223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:10.757661 3223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:10.758597 3223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:10.760023 3223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:10.760161 3223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:10.760314 3223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:10.760421 3223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:10.794631 3223 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:41951
I20260806 10:52:10.794683 3352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:41951 every 8 connection(s)
I20260806 10:52:10.795912 3223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:52:10.800877 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3223
I20260806 10:52:10.801048 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:52:10.801506 16468 external_mini_cluster.cc:1428] Setting key 1d901b8226096dec97d6f2b4ecf83066
I20260806 10:52:10.804600 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44447
--builtin_ntp_servers=127.16.21.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:52:10.808573 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:10.798445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:48777 (local address 127.16.21.62:44447)
0806 10:52:10.798695 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:10.798701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:10.800007 (+ 1306us) server_negotiation.cc:407] Connection header received
0806 10:52:10.801938 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:10.801943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:10.802027 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:10.802142 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:10.803448 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.804178 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.805720 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.806188 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.806328 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:10.807711 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:10.807717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:10.807726 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:10.807743 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:10.807778 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:10.807790 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:10.807889 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:10.808104 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 10:52:10.808199 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 10:52:10.809527 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:10.798963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44447 (local address 127.16.21.2:48777)
0806 10:52:10.799856 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:10.799892 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:10.801667 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:10.802422 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:10.802430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:10.802511 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:10.803246 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:10.803280 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.804360 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.804365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:10.805568 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:10.805580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:10.806415 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:10.806422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:10.806721 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:10.807408 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:10.807410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:10.807453 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:10.807521 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:10.807890 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:10.807904 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:10.807913 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:52:10.808092 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 10:52:10.808482 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":305,"threads_started":1}
I20260806 10:52:10.810894 3353 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44447
I20260806 10:52:10.811194 3353 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:10.811900 3353 heartbeater.cc:511] Master 127.16.21.62:44447 requested a full tablet report, sending...
I20260806 10:52:10.813273 3030 ts_manager.cc:194] Registered new tserver with Master: 9d471cb334964b84b952f16f84b81359 (127.16.21.2:41951)
I20260806 10:52:10.814078 3030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.2:48777
W20260806 10:52:11.143112 3356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:11.143515 3356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:11.143664 3356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:11.151098 3356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:11.151206 3356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:11.151309 3356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:52:11.160341 3356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:44447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:11.161485 3356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:11.163122 3356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:11.176437 3361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:11.176483 3362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:11.176502 3364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:11.177256 3356 server_base.cc:1034] running on GCE node
I20260806 10:52:11.177872 3356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:11.178821 3356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:11.180073 3356 hybrid_clock.cc:648] HybridClock initialized: now 1786013531180025 us; error 35 us; skew 500 ppm
I20260806 10:52:11.180471 3356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:11.182684 3356 webserver.cc:460] Webserver started at http://127.16.21.3:43913/ using document root <none> and password file <none>
I20260806 10:52:11.183377 3356 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:11.183517 3356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:11.183862 3356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:11.186854 3356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf"
format_stamp: "Formatted at 2026-08-06 10:52:11 on dist-test-slave-0b58"
server_key: "3921252a1bb31178532d9084f420d9e9"
server_key_iv: "9ee07d3996c8bae85670c773709784f0"
server_key_version: "encryptionkey@0"
I20260806 10:52:11.187681 3356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf"
format_stamp: "Formatted at 2026-08-06 10:52:11 on dist-test-slave-0b58"
server_key: "3921252a1bb31178532d9084f420d9e9"
server_key_iv: "9ee07d3996c8bae85670c773709784f0"
server_key_version: "encryptionkey@0"
I20260806 10:52:11.192826 3356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:52:11.196055 3370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:11.197347 3356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:11.197538 3356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf"
format_stamp: "Formatted at 2026-08-06 10:52:11 on dist-test-slave-0b58"
server_key: "3921252a1bb31178532d9084f420d9e9"
server_key_iv: "9ee07d3996c8bae85670c773709784f0"
server_key_version: "encryptionkey@0"
I20260806 10:52:11.197749 3356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:11.214939 3356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:11.215692 3356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:11.215991 3356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:11.216991 3356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:11.218459 3356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:11.218590 3356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:11.218731 3356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:11.218829 3356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:11.254009 3356 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:41589
I20260806 10:52:11.254056 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:41589 every 8 connection(s)
I20260806 10:52:11.255271 3356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:52:11.258440 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3356
I20260806 10:52:11.258638 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:52:11.259183 16468 external_mini_cluster.cc:1428] Setting key 130b0f0031993b527907baaede0af3c3
I20260806 10:52:11.267674 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.257722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:46357 (local address 127.16.21.62:44447)
0806 10:52:11.257901 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.257906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.259463 (+ 1557us) server_negotiation.cc:407] Connection header received
0806 10:52:11.261721 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.261725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.261795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.261873 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:11.263091 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.263809 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.265347 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.265727 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.265825 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.267038 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:11.267043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:11.267046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:11.267063 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:11.267092 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:11.267103 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:11.267170 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.267377 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.267449 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:52:11.268671 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.258240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44447 (local address 127.16.21.3:46357)
0806 10:52:11.259295 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:11.259341 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:11.261454 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:11.262155 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:11.262163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:11.262245 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:11.262942 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.262960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.263945 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.263949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.265223 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.265236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.265918 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.265922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.266117 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.266760 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:11.266763 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:11.266805 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:11.266879 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:11.267190 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:11.267198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:11.267203 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:52:11.267333 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:52:11.267594 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":335,"threads_started":1}
I20260806 10:52:11.270005 3485 heartbeater.cc:347] Connected to a master server at 127.16.21.62:44447
I20260806 10:52:11.270299 3485 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:11.270982 3485 heartbeater.cc:511] Master 127.16.21.62:44447 requested a full tablet report, sending...
I20260806 10:52:11.272332 3030 ts_manager.cc:194] Registered new tserver with Master: 99771e7ea1b84f3fb56f161a5ebdb8bf (127.16.21.3:41589)
I20260806 10:52:11.273288 3030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.3:46357
I20260806 10:52:11.274616 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:52:11.284788 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.278689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.16.21.62:44447)
0806 10:52:11.278842 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.278846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.279356 (+ 510us) server_negotiation.cc:407] Connection header received
0806 10:52:11.279568 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.279574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.279652 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.279761 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:11.280328 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.280987 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.282357 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.282733 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.282868 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.284030 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:11.284035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:11.284038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:11.284055 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:11.284084 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:11.284095 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:11.284248 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.284484 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.284538 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:52:11.288657 3030 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:52:11.295550 3030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:52:11.329012 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.307376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.16.21.1:40875)
0806 10:52:11.320122 (+ 12746us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.320131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.320160 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:52:11.320255 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.320311 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.320559 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.320763 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:11.321502 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.323372 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.326469 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.327789 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.327935 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.328188 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.328495 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.328651 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12581,"thread_start_us":13583,"threads_started":1}
I20260806 10:52:11.329022 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.307221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40875 (local address 127.0.0.1:57578)
0806 10:52:11.318687 (+ 11466us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:11.318735 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:11.318965 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:11.320774 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:11.320781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:11.320810 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:11.321249 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.321266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.323545 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.323550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.326098 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.326109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.327987 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.327994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.328207 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.328324 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 10:52:11.328454 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:52:11.328648 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11149,"mutex_wait_us":27,"thread_start_us":11933,"threads_started":1}
I20260806 10:52:11.331172 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.307996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.16.21.3:41589)
0806 10:52:11.322611 (+ 14615us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.322618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.322649 (+ 31us) server_negotiation.cc:407] Connection header received
0806 10:52:11.322737 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.322746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.322955 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.323885 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:11.324147 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.325718 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.328789 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.330194 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.330425 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.330698 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.330779 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.330912 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14444,"thread_start_us":14306,"threads_started":1}
I20260806 10:52:11.331209 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.307370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41589 (local address 127.0.0.1:43690)
0806 10:52:11.321562 (+ 14192us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:11.321583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:11.321683 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:11.323141 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:11.323150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:11.323198 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:11.323694 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.323707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.325866 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.325871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.328555 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.328571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.330436 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.330446 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.330624 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.330654 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:52:11.330745 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:52:11.330865 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14055,"mutex_wait_us":158,"spinlock_wait_cycles":19328,"thread_start_us":13921,"threads_started":1}
I20260806 10:52:11.332345 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.307919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:41951 (local address 127.0.0.1:38624)
0806 10:52:11.322975 (+ 15056us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:11.322992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:11.323198 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:11.324005 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:11.324008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:11.324027 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:11.324544 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.324554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.325852 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.325857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.328587 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.328600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.331552 (+ 2952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.331561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.331712 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.331741 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 10:52:11.331822 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 10:52:11.331935 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14906,"mutex_wait_us":269,"spinlock_wait_cycles":1280,"thread_start_us":14751,"threads_started":1}
I20260806 10:52:11.332468 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.308548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.16.21.2:41951)
0806 10:52:11.322133 (+ 13585us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.322141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.323080 (+ 939us) server_negotiation.cc:407] Connection header received
0806 10:52:11.323400 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.323407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.323613 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.323782 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:11.324749 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.325698 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.328797 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.330194 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.330350 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.330751 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.331919 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.332106 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13432,"thread_start_us":13335,"threads_started":1}
I20260806 10:52:11.333583 3152 tablet_service.cc:1467] Processing CreateTablet for tablet 32d330b48c754a6db68366f2738227b3 (DEFAULT_TABLE table=test-table [id=027e6b7637e9410e9a9fe304fd610e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:11.334844 3152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d330b48c754a6db68366f2738227b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:11.334761 3418 tablet_service.cc:1467] Processing CreateTablet for tablet 32d330b48c754a6db68366f2738227b3 (DEFAULT_TABLE table=test-table [id=027e6b7637e9410e9a9fe304fd610e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:11.336143 3418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d330b48c754a6db68366f2738227b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:11.336236 3286 tablet_service.cc:1467] Processing CreateTablet for tablet 32d330b48c754a6db68366f2738227b3 (DEFAULT_TABLE table=test-table [id=027e6b7637e9410e9a9fe304fd610e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:11.337533 3286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d330b48c754a6db68366f2738227b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:11.353564 3504 tablet_bootstrap.cc:492] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Bootstrap starting.
I20260806 10:52:11.360136 3504 tablet_bootstrap.cc:654] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:11.361037 3503 tablet_bootstrap.cc:492] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Bootstrap starting.
I20260806 10:52:11.361905 3504 log.cc:824] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:11.364332 3503 tablet_bootstrap.cc:654] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:11.365518 3503 log.cc:824] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:11.365639 3504 tablet_bootstrap.cc:492] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: No bootstrap required, opened a new log
I20260806 10:52:11.365960 3504 ts_tablet_manager.cc:1397] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260806 10:52:11.366124 3219 heartbeater.cc:503] Master 127.16.21.62:44447 was elected leader, sending a full tablet report...
I20260806 10:52:11.366747 3505 tablet_bootstrap.cc:492] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Bootstrap starting.
I20260806 10:52:11.370182 3505 tablet_bootstrap.cc:654] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:11.371656 3505 log.cc:824] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:11.374624 3503 tablet_bootstrap.cc:492] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: No bootstrap required, opened a new log
I20260806 10:52:11.375128 3503 ts_tablet_manager.cc:1397] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260806 10:52:11.375204 3505 tablet_bootstrap.cc:492] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: No bootstrap required, opened a new log
I20260806 10:52:11.375497 3505 ts_tablet_manager.cc:1397] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260806 10:52:11.375340 3504 raft_consensus.cc:348] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.375968 3504 raft_consensus.cc:374] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:11.376123 3504 raft_consensus.cc:729] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99771e7ea1b84f3fb56f161a5ebdb8bf, State: Initialized, Role: FOLLOWER
I20260806 10:52:11.376786 3504 consensus_queue.cc:260] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.377997 3485 heartbeater.cc:503] Master 127.16.21.62:44447 was elected leader, sending a full tablet report...
I20260806 10:52:11.378674 3504 ts_tablet_manager.cc:1428] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260806 10:52:11.386245 3505 raft_consensus.cc:348] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.386242 3503 raft_consensus.cc:348] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.386688 3503 raft_consensus.cc:374] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:11.386686 3505 raft_consensus.cc:374] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:11.386796 3505 raft_consensus.cc:729] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d471cb334964b84b952f16f84b81359, State: Initialized, Role: FOLLOWER
I20260806 10:52:11.386796 3503 raft_consensus.cc:729] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3871c452af64a33aaf63ca98cae7ffc, State: Initialized, Role: FOLLOWER
I20260806 10:52:11.387419 3503 consensus_queue.cc:260] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.387418 3505 consensus_queue.cc:260] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.392339 3353 heartbeater.cc:503] Master 127.16.21.62:44447 was elected leader, sending a full tablet report...
I20260806 10:52:11.393138 3505 ts_tablet_manager.cc:1428] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260806 10:52:11.393455 3503 ts_tablet_manager.cc:1428] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Time spent starting tablet: real 0.018s user 0.008s sys 0.009s
I20260806 10:52:11.473724 3510 raft_consensus.cc:482] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:52:11.474105 3510 raft_consensus.cc:504] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.475898 3510 leader_election.cc:288] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3871c452af64a33aaf63ca98cae7ffc (127.16.21.1:40875), 99771e7ea1b84f3fb56f161a5ebdb8bf (127.16.21.3:41589)
I20260806 10:52:11.482998 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.476214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:37953 (local address 127.16.21.1:40875)
0806 10:52:11.476461 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.476467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.477443 (+ 976us) server_negotiation.cc:407] Connection header received
0806 10:52:11.477726 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.477731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.477817 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.477919 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:11.478659 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.479348 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.481274 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.482044 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.482194 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.482289 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.482617 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.482738 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 10:52:11.483064 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.476338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:40875 (local address 127.16.21.2:37953)
0806 10:52:11.477310 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:11.477338 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:11.477573 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:11.477945 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:11.477950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:11.477974 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:11.478495 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.478511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.479506 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.479515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.481078 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.481090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.482242 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.482248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.482443 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.482469 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:52:11.482580 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:52:11.482703 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":53,"thread_start_us":539,"threads_started":1}
I20260806 10:52:11.483234 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.476460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:41589 (local address 127.16.21.2:48323)
0806 10:52:11.477530 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:11.477546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:11.477713 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:11.478138 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:11.478143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:11.478173 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:11.478728 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.478740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.479726 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.479731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.481202 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:11.481214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.482399 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.482405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:11.482593 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.482616 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:52:11.482735 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 10:52:11.482855 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":8,"thread_start_us":750,"threads_started":1}
I20260806 10:52:11.483397 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.476600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:48323 (local address 127.16.21.3:41589)
0806 10:52:11.476838 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.476847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.477617 (+ 770us) server_negotiation.cc:407] Connection header received
0806 10:52:11.477898 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.477906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.477987 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.478086 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:11.478872 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.479565 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.481391 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.482195 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.482345 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.482449 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.482762 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.482882 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:52:11.484061 3172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d330b48c754a6db68366f2738227b3" candidate_uuid: "9d471cb334964b84b952f16f84b81359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3871c452af64a33aaf63ca98cae7ffc" is_pre_election: true
I20260806 10:52:11.484563 3172 raft_consensus.cc:2454] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d471cb334964b84b952f16f84b81359 in term 0.
I20260806 10:52:11.484467 3438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d330b48c754a6db68366f2738227b3" candidate_uuid: "9d471cb334964b84b952f16f84b81359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" is_pre_election: true
I20260806 10:52:11.484951 3438 raft_consensus.cc:2454] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d471cb334964b84b952f16f84b81359 in term 0.
I20260806 10:52:11.485247 3238 leader_election.cc:302] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d471cb334964b84b952f16f84b81359, f3871c452af64a33aaf63ca98cae7ffc; no voters:
I20260806 10:52:11.485720 3510 raft_consensus.cc:2781] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:52:11.485868 3510 raft_consensus.cc:482] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:52:11.485992 3510 raft_consensus.cc:3037] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:11.488220 3510 raft_consensus.cc:504] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.488852 3510 leader_election.cc:288] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [CANDIDATE]: Term 1 election: Requested vote from peers f3871c452af64a33aaf63ca98cae7ffc (127.16.21.1:40875), 99771e7ea1b84f3fb56f161a5ebdb8bf (127.16.21.3:41589)
I20260806 10:52:11.489291 3172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d330b48c754a6db68366f2738227b3" candidate_uuid: "9d471cb334964b84b952f16f84b81359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3871c452af64a33aaf63ca98cae7ffc"
I20260806 10:52:11.489339 3438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d330b48c754a6db68366f2738227b3" candidate_uuid: "9d471cb334964b84b952f16f84b81359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf"
I20260806 10:52:11.489492 3172 raft_consensus.cc:3037] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:11.489549 3438 raft_consensus.cc:3037] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:11.491784 3438 raft_consensus.cc:2454] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d471cb334964b84b952f16f84b81359 in term 1.
I20260806 10:52:11.491816 3172 raft_consensus.cc:2454] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d471cb334964b84b952f16f84b81359 in term 1.
I20260806 10:52:11.492192 3239 leader_election.cc:302] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99771e7ea1b84f3fb56f161a5ebdb8bf, 9d471cb334964b84b952f16f84b81359; no voters:
I20260806 10:52:11.492615 3510 raft_consensus.cc:2781] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:11.493072 3510 raft_consensus.cc:686] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 1 LEADER]: Becoming Leader. State: Replica: 9d471cb334964b84b952f16f84b81359, State: Running, Role: LEADER
I20260806 10:52:11.493522 3510 consensus_queue.cc:237] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } }
I20260806 10:52:11.497753 3030 catalog_manager.cc:5495] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d471cb334964b84b952f16f84b81359 (127.16.21.2). New cstate: current_term: 1 leader_uuid: "9d471cb334964b84b952f16f84b81359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d471cb334964b84b952f16f84b81359" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 41951 } health_report { overall_health: HEALTHY } } }
W20260806 10:52:11.506970 3486 tablet.cc:2369] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:11.526726 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.521449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.16.21.2:41951)
0806 10:52:11.521653 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.521661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.521775 (+ 114us) server_negotiation.cc:407] Connection header received
0806 10:52:11.521923 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.521926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.522007 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.522121 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:52:11.522468 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.523144 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.524616 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.525018 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.525110 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.525622 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:52:11.526067 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:52:11.526218 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.526361 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.526430 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:52:11.534495 3172 raft_consensus.cc:1261] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 1 FOLLOWER]: Refusing update from remote peer 9d471cb334964b84b952f16f84b81359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:11.534533 3438 raft_consensus.cc:1261] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 1 FOLLOWER]: Refusing update from remote peer 9d471cb334964b84b952f16f84b81359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:11.535460 3514 consensus_queue.cc:1035] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3871c452af64a33aaf63ca98cae7ffc" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 40875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:11.535794 3510 consensus_queue.cc:1035] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99771e7ea1b84f3fb56f161a5ebdb8bf" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:11.545207 3517 mvcc.cc:205] Tried to move back new op lower bound from 7315511425155493888 to 7315511425001197568. Current Snapshot: MvccSnapshot[applied={T|T < 7315511425155493888}]
I20260806 10:52:11.562026 3029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:11.562286 3029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:11.565716 3029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa1c13c1fd6440459c8dd6e9b5f612e3: Sending DeleteTablet for 3 replicas of tablet 32d330b48c754a6db68366f2738227b3
I20260806 10:52:11.566821 3152 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d330b48c754a6db68366f2738227b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:11 UTC) from {username='slave'} at 127.0.0.1:57578
I20260806 10:52:11.566918 3418 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d330b48c754a6db68366f2738227b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:11 UTC) from {username='slave'} at 127.0.0.1:43690
I20260806 10:52:11.566931 3286 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d330b48c754a6db68366f2738227b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:11 UTC) from {username='slave'} at 127.0.0.1:38624
I20260806 10:52:11.570017 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3090
I20260806 10:52:11.579200 3527 tablet_replica.cc:330] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: stopping tablet replica
I20260806 10:52:11.579331 3529 tablet_replica.cc:330] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: stopping tablet replica
I20260806 10:52:11.579909 3527 raft_consensus.cc:2229] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:11.580227 3529 raft_consensus.cc:2229] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:52:11.580502 3527 raft_consensus.cc:2258] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:11.581421 3529 raft_consensus.cc:2258] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:11.582864 3527 ts_tablet_manager.cc:1905] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:11.584229 3528 tablet_replica.cc:330] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: stopping tablet replica
I20260806 10:52:11.584728 3528 raft_consensus.cc:2229] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:11.585269 3528 raft_consensus.cc:2258] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:11.587235 3528 ts_tablet_manager.cc:1905] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:11.591096 3527 ts_tablet_manager.cc:1918] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:11.591316 3527 log.cc:1196] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/32d330b48c754a6db68366f2738227b3
I20260806 10:52:11.592413 3527 ts_tablet_manager.cc:1939] T 32d330b48c754a6db68366f2738227b3 P f3871c452af64a33aaf63ca98cae7ffc: Deleting consensus metadata
I20260806 10:52:11.593035 3529 ts_tablet_manager.cc:1905] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:11.594362 3014 catalog_manager.cc:4841] TS f3871c452af64a33aaf63ca98cae7ffc (127.16.21.1:40875): tablet 32d330b48c754a6db68366f2738227b3 (table test-table [id=027e6b7637e9410e9a9fe304fd610e28]) successfully deleted
I20260806 10:52:11.594483 3528 ts_tablet_manager.cc:1918] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:11.594620 3528 log.cc:1196] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/32d330b48c754a6db68366f2738227b3
I20260806 10:52:11.595220 3528 ts_tablet_manager.cc:1939] T 32d330b48c754a6db68366f2738227b3 P 99771e7ea1b84f3fb56f161a5ebdb8bf: Deleting consensus metadata
I20260806 10:52:11.596639 3015 catalog_manager.cc:4841] TS 99771e7ea1b84f3fb56f161a5ebdb8bf (127.16.21.3:41589): tablet 32d330b48c754a6db68366f2738227b3 (table test-table [id=027e6b7637e9410e9a9fe304fd610e28]) successfully deleted
I20260806 10:52:11.599479 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.571438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.16.21.1:40875)
0806 10:52:11.571618 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.571624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.585040 (+ 13416us) server_negotiation.cc:407] Connection header received
0806 10:52:11.585482 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.585488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.585602 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.585681 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:11.586881 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.588083 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.590085 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.590767 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.590867 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.598100 (+ 7233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:11.598109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:11.598121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:11.598656 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:11.598700 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:11.598719 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:11.598858 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.599006 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.599114 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:52:11.600167 3214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:11.600736 3529 ts_tablet_manager.cc:1918] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:11.601042 3529 log.cc:1196] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/32d330b48c754a6db68366f2738227b3
I20260806 10:52:11.601879 3529 ts_tablet_manager.cc:1939] T 32d330b48c754a6db68366f2738227b3 P 9d471cb334964b84b952f16f84b81359: Deleting consensus metadata
I20260806 10:52:11.603711 3016 catalog_manager.cc:4841] TS 9d471cb334964b84b952f16f84b81359 (127.16.21.2:41951): tablet 32d330b48c754a6db68366f2738227b3 (table test-table [id=027e6b7637e9410e9a9fe304fd610e28]) successfully deleted
I20260806 10:52:11.689559 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3090
I20260806 10:52:11.717206 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3223
I20260806 10:52:11.725101 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.718070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.16.21.2:41951)
0806 10:52:11.718247 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.718251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.718273 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:52:11.718489 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.718493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.718572 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.718658 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:11.719115 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.719921 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.721857 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.722271 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.722376 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.723523 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:11.723532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:11.723545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:11.724133 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:11.724180 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:11.724201 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:11.724526 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.724673 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.724779 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:52:11.725817 3347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:11.837858 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3223
I20260806 10:52:11.860229 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3356
I20260806 10:52:11.866750 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:11.861066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.16.21.3:41589)
0806 10:52:11.861261 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:11.861266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:11.861301 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:52:11.861475 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:11.861479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:11.861562 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:11.861646 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:11.862010 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.862732 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.863941 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:11.864438 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:11.864560 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:11.865584 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:11.865591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:11.865605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:11.866149 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:11.866213 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:11.866232 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:11.866316 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:11.866440 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 10:52:11.866523 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:52:11.867381 3479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:11.957249 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3356
I20260806 10:52:11.977154 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 2999
I20260806 10:52:11.978248 3060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:12.089825 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 2999
2026-08-06T10:52:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:12 dist-test-slave-0b58 krb5kdc[3537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:12 dist-test-slave-0b58 krb5kdc[3537](info): set up 2 sockets
Aug 06 10:52:12 dist-test-slave-0b58 krb5kdc[3537](info): commencing operation
krb5kdc: starting...
W20260806 10:52:13.243980 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:13 dist-test-slave-0b58 krb5kdc[3537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:13.551692 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36637
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:36637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:13.889067 3550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:13.889411 3550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:13.889577 3550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:13.896879 3550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:13.897027 3550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:13.897140 3550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:13.897253 3550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:13.897338 3550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:13.907114 3550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:36637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:13.908880 3550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:13.910158 3550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:13.922029 3555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:13.922101 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:13.922057 3558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:13.923022 3550 server_base.cc:1034] running on GCE node
I20260806 10:52:13.923748 3550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:13.925138 3550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:13.926379 3550 hybrid_clock.cc:648] HybridClock initialized: now 1786013533926338 us; error 57 us; skew 500 ppm
I20260806 10:52:13.926718 3550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:13 dist-test-slave-0b58 krb5kdc[3537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013533, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:13.929860 3550 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:13.931257 3550 webserver.cc:460] Webserver started at http://127.16.21.62:35399/ using document root <none> and password file <none>
I20260806 10:52:13.931931 3550 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:13.932088 3550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:13.932473 3550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:13.935365 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "20c82b8c42604c2cb63113d4aecd1e9b"
format_stamp: "Formatted at 2026-08-06 10:52:13 on dist-test-slave-0b58"
server_key: "68af62bd9bc0f2e352eb42516355cdb1"
server_key_iv: "101070b6d22e2c6a9a793fe18e063326"
server_key_version: "encryptionkey@0"
I20260806 10:52:13.936131 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "20c82b8c42604c2cb63113d4aecd1e9b"
format_stamp: "Formatted at 2026-08-06 10:52:13 on dist-test-slave-0b58"
server_key: "68af62bd9bc0f2e352eb42516355cdb1"
server_key_iv: "101070b6d22e2c6a9a793fe18e063326"
server_key_version: "encryptionkey@0"
I20260806 10:52:13.941285 3550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 10:52:13.944756 3565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:13.946204 3550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:52:13.946401 3550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "20c82b8c42604c2cb63113d4aecd1e9b"
format_stamp: "Formatted at 2026-08-06 10:52:13 on dist-test-slave-0b58"
server_key: "68af62bd9bc0f2e352eb42516355cdb1"
server_key_iv: "101070b6d22e2c6a9a793fe18e063326"
server_key_version: "encryptionkey@0"
I20260806 10:52:13.946609 3550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:13.982705 3550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:13.986635 3550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:13.986939 3550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:14.005475 3550 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:36637
I20260806 10:52:14.005522 3618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:36637 every 8 connection(s)
I20260806 10:52:14.006855 3550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:14.009660 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3550
I20260806 10:52:14.009853 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:14.010432 16468 external_mini_cluster.cc:1428] Setting key 42854897b1ead8c978c1687b497fe79b
I20260806 10:52:14.011072 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:14.023079 3619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b: Bootstrap starting.
Aug 06 10:52:14 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:14.026265 3619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:14.027411 3619 log.cc:824] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:14.030818 3619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b: No bootstrap required, opened a new log
I20260806 10:52:14.034206 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:14.012422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.16.21.62:36637)
0806 10:52:14.013224 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:14.013234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:14.013272 (+ 38us) server_negotiation.cc:407] Connection header received
0806 10:52:14.015766 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:14.015787 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:14.016146 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:14.016653 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:14.018205 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:14.019473 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:14.020851 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:14.021574 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:14.021775 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:14.026192 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:14.026200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:14.026213 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:14.026257 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:14.029912 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:14.030790 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:14.030800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:14.030806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:14.031078 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:14.031538 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:14.031541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:14.031544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:14.032019 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:14.032250 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:14.032811 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 10:52:14.033191 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":357,"threads_started":1}
I20260806 10:52:14.039518 3619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36637 } }
I20260806 10:52:14.040002 3619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:14.040118 3619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c82b8c42604c2cb63113d4aecd1e9b, State: Initialized, Role: FOLLOWER
I20260806 10:52:14.040735 3619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36637 } }
I20260806 10:52:14.040977 3619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:14.041127 3619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:14.041302 3619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:14.043324 3619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36637 } }
I20260806 10:52:14.043812 3619 leader_election.cc:302] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20c82b8c42604c2cb63113d4aecd1e9b; no voters:
I20260806 10:52:14.044454 3619 leader_election.cc:288] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:14.044596 3624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:14.045913 3624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [term 1 LEADER]: Becoming Leader. State: Replica: 20c82b8c42604c2cb63113d4aecd1e9b, State: Running, Role: LEADER
I20260806 10:52:14.046567 3619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:14.046520 3624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36637 } }
I20260806 10:52:14.049561 3625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36637 } } }
I20260806 10:52:14.049960 3625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:14.050463 3626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20c82b8c42604c2cb63113d4aecd1e9b. Latest consensus state: current_term: 1 leader_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20c82b8c42604c2cb63113d4aecd1e9b" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36637 } } }
I20260806 10:52:14.050763 3626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:14.051964 3633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:14.057129 3633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:14.065683 3633 catalog_manager.cc:1347] Generated new cluster ID: 19374cfb5441431abb5c143a7d10b923
I20260806 10:52:14.065806 3633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:14.082403 3633 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:14.084126 3633 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:14.094157 3633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b: Generated new TSK 0
I20260806 10:52:14.094894 3633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
Entry for principal kudu/127.16.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab.
I20260806 10:52:14.180325 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36637
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:14.527347 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:14.527649 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:14.527810 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:14.535264 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:14.535404 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:14.535596 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:52:14.545641 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.1.keytab
--principal=kudu/127.16.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:14.547101 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:14.548602 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:14.562036 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:14.562074 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:14.562036 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:14.563093 3645 server_base.cc:1034] running on GCE node
I20260806 10:52:14.563670 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:14.564657 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:14.565891 3645 hybrid_clock.cc:648] HybridClock initialized: now 1786013534565855 us; error 49 us; skew 500 ppm
I20260806 10:52:14.566216 3645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:14 dist-test-slave-0b58 krb5kdc[3537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013534, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:14.569319 3645 init.cc:377] Logged in from keytab as kudu/127.16.21.1@KRBTEST.COM (short username kudu)
I20260806 10:52:14.570796 3645 webserver.cc:460] Webserver started at http://127.16.21.1:35935/ using document root <none> and password file <none>
I20260806 10:52:14.571453 3645 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:14.571609 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:14.571930 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:14.574832 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "fc9daf7b7afc41f2a3484a70be691309"
format_stamp: "Formatted at 2026-08-06 10:52:14 on dist-test-slave-0b58"
server_key: "f15dfef40c63de5788362e66680f3b54"
server_key_iv: "d61d25bc29edf43f82de9f3300917561"
server_key_version: "encryptionkey@0"
I20260806 10:52:14.575605 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9daf7b7afc41f2a3484a70be691309"
format_stamp: "Formatted at 2026-08-06 10:52:14 on dist-test-slave-0b58"
server_key: "f15dfef40c63de5788362e66680f3b54"
server_key_iv: "d61d25bc29edf43f82de9f3300917561"
server_key_version: "encryptionkey@0"
I20260806 10:52:14.580615 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:52:14.584007 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:14.585332 3645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:14.585501 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "fc9daf7b7afc41f2a3484a70be691309"
format_stamp: "Formatted at 2026-08-06 10:52:14 on dist-test-slave-0b58"
server_key: "f15dfef40c63de5788362e66680f3b54"
server_key_iv: "d61d25bc29edf43f82de9f3300917561"
server_key_version: "encryptionkey@0"
I20260806 10:52:14.585697 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:14.603946 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:14.607734 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:14.608026 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:14.609009 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:14.610316 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:14.610422 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:14.610585 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:14.610663 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:14.644716 3645 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:37655
I20260806 10:52:14.644765 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:37655 every 8 connection(s)
I20260806 10:52:14.646000 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:52:14.654431 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3645
I20260806 10:52:14.654611 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:52:14.655067 16468 external_mini_cluster.cc:1428] Setting key db77d4de2649f47da21c044c4225117e
Aug 06 10:52:14 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013534, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.1@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:14.666428 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:14.648785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:43315 (local address 127.16.21.62:36637)
0806 10:52:14.649234 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:14.649238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:14.650294 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 10:52:14.652202 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:14.652206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:14.652340 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:14.652442 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:14.653867 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:14.654651 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:14.656475 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:14.657004 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:14.657143 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:14.660828 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:14.660834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:14.660839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:14.660880 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:14.663349 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:14.664062 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:14.664067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:14.664071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:14.664204 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:14.664709 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:14.664714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:14.664718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:14.664990 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:14.665149 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:14.665816 (+ 667us) server_negotiation.cc:299] Negotiation successful
0806 10:52:14.666037 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260806 10:52:14.667318 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:14.649370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36637 (local address 127.16.21.1:43315)
0806 10:52:14.650066 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:14.650120 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:14.651963 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:14.652689 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:14.652697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:14.653181 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:14.653716 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:14.653732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:14.654855 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:14.654859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:14.656331 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:14.656344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:14.657233 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:14.657242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:14.657499 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:14.658190 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:14.658217 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:14.660611 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:14.663497 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:14.663505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:14.663519 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:14.663927 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:14.664372 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:14.664377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:14.664381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:14.664576 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:14.665135 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:14.665142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:14.665450 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 10:52:14.665717 (+ 267us) client_negotiation.cc:240] Negotiation successful
0806 10:52:14.666241 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":288,"threads_started":1}
I20260806 10:52:14.668795 3774 heartbeater.cc:347] Connected to a master server at 127.16.21.62:36637
I20260806 10:52:14.669091 3774 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:14.669786 3774 heartbeater.cc:511] Master 127.16.21.62:36637 requested a full tablet report, sending...
I20260806 10:52:14.671811 3582 ts_manager.cc:194] Registered new tserver with Master: fc9daf7b7afc41f2a3484a70be691309 (127.16.21.1:37655)
I20260806 10:52:14.673309 3582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.1@KRBTEST.COM'} at 127.16.21.1:43315
WARNING: no policy specified for kudu/127.16.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
Entry for principal kudu/127.16.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab.
I20260806 10:52:14.727003 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36637
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:15.067435 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:15.067703 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:15.067795 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:15.074899 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:15.074994 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:15.075095 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:52:15.084595 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.2.keytab
--principal=kudu/127.16.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:15.085762 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:15.086988 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:15.100507 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:15.100576 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:15.100322 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:15.105633 3780 server_base.cc:1034] running on GCE node
I20260806 10:52:15.106262 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:15.107180 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:15.108423 3780 hybrid_clock.cc:648] HybridClock initialized: now 1786013535108395 us; error 61 us; skew 500 ppm
I20260806 10:52:15.108748 3780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:15 dist-test-slave-0b58 krb5kdc[3537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013535, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:15.111970 3780 init.cc:377] Logged in from keytab as kudu/127.16.21.2@KRBTEST.COM (short username kudu)
I20260806 10:52:15.113412 3780 webserver.cc:460] Webserver started at http://127.16.21.2:39579/ using document root <none> and password file <none>
I20260806 10:52:15.114050 3780 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:15.114149 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:15.114411 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:15.117338 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "0f34131544294f199eb7ca775f977d0d"
format_stamp: "Formatted at 2026-08-06 10:52:15 on dist-test-slave-0b58"
server_key: "9ff71b91cc4a06f833603a9db732023d"
server_key_iv: "3498e3537c474d7f55fe5e0da3ae0f00"
server_key_version: "encryptionkey@0"
I20260806 10:52:15.118045 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "0f34131544294f199eb7ca775f977d0d"
format_stamp: "Formatted at 2026-08-06 10:52:15 on dist-test-slave-0b58"
server_key: "9ff71b91cc4a06f833603a9db732023d"
server_key_iv: "3498e3537c474d7f55fe5e0da3ae0f00"
server_key_version: "encryptionkey@0"
I20260806 10:52:15.123005 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 10:52:15.126420 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:15.127595 3780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:52:15.127764 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "0f34131544294f199eb7ca775f977d0d"
format_stamp: "Formatted at 2026-08-06 10:52:15 on dist-test-slave-0b58"
server_key: "9ff71b91cc4a06f833603a9db732023d"
server_key_iv: "3498e3537c474d7f55fe5e0da3ae0f00"
server_key_version: "encryptionkey@0"
I20260806 10:52:15.127976 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:15.147666 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:15.152216 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:15.152573 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:15.153509 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:15.154811 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:15.154884 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:15.155062 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:15.155115 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:15.189522 3780 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:42805
I20260806 10:52:15.189579 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:42805 every 8 connection(s)
I20260806 10:52:15.190896 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:52:15.196571 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3780
I20260806 10:52:15.196779 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:52:15.197356 16468 external_mini_cluster.cc:1428] Setting key b5dd31bbe6602cd2194a10b79d182817
Aug 06 10:52:15 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013535, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.2@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:15.212594 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.193489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:38435 (local address 127.16.21.62:36637)
0806 10:52:15.193970 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:15.193976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:15.194943 (+ 967us) server_negotiation.cc:407] Connection header received
0806 10:52:15.196863 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:15.196869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:15.196970 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:15.197094 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:15.198636 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.199636 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.201253 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.202042 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.202184 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.205674 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:15.205682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:15.205689 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:15.205734 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:15.209347 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:15.210039 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:15.210047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:15.210053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:15.210216 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:15.210616 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:15.210624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:15.210629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:15.211013 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:15.211173 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:15.211817 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 10:52:15.212057 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260806 10:52:15.213172 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.194085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36637 (local address 127.16.21.2:38435)
0806 10:52:15.194775 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:15.194817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:15.196568 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:15.197286 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:15.197294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:15.197853 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:15.198465 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.198489 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.199809 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.199812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.201106 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.201118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.202252 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.202257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.202451 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.203143 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:15.203169 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:15.205433 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:15.209483 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:15.209489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:15.209499 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:15.209904 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:15.210327 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:15.210330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:15.210332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:15.210498 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:15.211132 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:15.211137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:15.211424 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 10:52:15.211662 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 10:52:15.212004 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":274,"threads_started":1}
I20260806 10:52:15.214653 3911 heartbeater.cc:347] Connected to a master server at 127.16.21.62:36637
I20260806 10:52:15.214951 3911 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:15.215648 3911 heartbeater.cc:511] Master 127.16.21.62:36637 requested a full tablet report, sending...
I20260806 10:52:15.217110 3582 ts_manager.cc:194] Registered new tserver with Master: 0f34131544294f199eb7ca775f977d0d (127.16.21.2:42805)
I20260806 10:52:15.217931 3582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.2@KRBTEST.COM'} at 127.16.21.2:38435
WARNING: no policy specified for kudu/127.16.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
Entry for principal kudu/127.16.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab.
I20260806 10:52:15.271512 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36637
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:15.611264 3917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:15.611515 3917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:15.611608 3917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:15.618719 3917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:15.618824 3917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:15.618935 3917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:52:15.628707 3917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.3.keytab
--principal=kudu/127.16.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:36637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:15.629850 3917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:15.631356 3917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:15.644667 3925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:15.644644 3923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:15.645385 3917 server_base.cc:1034] running on GCE node
W20260806 10:52:15.648819 3922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:15.649613 3917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:15.650667 3917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:15.651890 3917 hybrid_clock.cc:648] HybridClock initialized: now 1786013535651815 us; error 73 us; skew 500 ppm
I20260806 10:52:15.652215 3917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:15 dist-test-slave-0b58 krb5kdc[3537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013535, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:15.655280 3917 init.cc:377] Logged in from keytab as kudu/127.16.21.3@KRBTEST.COM (short username kudu)
I20260806 10:52:15.656678 3917 webserver.cc:460] Webserver started at http://127.16.21.3:43075/ using document root <none> and password file <none>
I20260806 10:52:15.657270 3917 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:15.657379 3917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:15.657632 3917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:15.660447 3917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "17e7399d24574b9db554864df9ad472e"
format_stamp: "Formatted at 2026-08-06 10:52:15 on dist-test-slave-0b58"
server_key: "011c104bb593fb7b4ed0d52bc9c1c3a9"
server_key_iv: "78e97edbb92ebcd277e34b413d39c235"
server_key_version: "encryptionkey@0"
I20260806 10:52:15.661155 3917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "17e7399d24574b9db554864df9ad472e"
format_stamp: "Formatted at 2026-08-06 10:52:15 on dist-test-slave-0b58"
server_key: "011c104bb593fb7b4ed0d52bc9c1c3a9"
server_key_iv: "78e97edbb92ebcd277e34b413d39c235"
server_key_version: "encryptionkey@0"
I20260806 10:52:15.666111 3917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:52:15.669479 3932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:15.670732 3917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:52:15.670904 3917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "17e7399d24574b9db554864df9ad472e"
format_stamp: "Formatted at 2026-08-06 10:52:15 on dist-test-slave-0b58"
server_key: "011c104bb593fb7b4ed0d52bc9c1c3a9"
server_key_iv: "78e97edbb92ebcd277e34b413d39c235"
server_key_version: "encryptionkey@0"
I20260806 10:52:15.671110 3917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:15.676034 3774 heartbeater.cc:503] Master 127.16.21.62:36637 was elected leader, sending a full tablet report...
I20260806 10:52:15.692646 3917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:15.696623 3917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:15.696907 3917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:15.697837 3917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:15.699090 3917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:15.699165 3917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:15.699347 3917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:15.699401 3917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:15.733198 3917 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:33299
I20260806 10:52:15.733287 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:33299 every 8 connection(s)
I20260806 10:52:15.734517 3917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:52:15.742743 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 3917
I20260806 10:52:15.742934 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:52:15.743407 16468 external_mini_cluster.cc:1428] Setting key 2b363a619fb9d15164faff01e3ebe983
Aug 06 10:52:15 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013535, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.3@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:15.755170 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.737104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:58471 (local address 127.16.21.62:36637)
0806 10:52:15.737597 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:15.737601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:15.738351 (+ 750us) server_negotiation.cc:407] Connection header received
0806 10:52:15.740191 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:15.740197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:15.740344 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:15.740473 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:15.741904 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.742961 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.744838 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.745258 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.745419 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.749266 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:15.749271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:15.749282 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:15.749313 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:15.752052 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:15.752846 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:15.752859 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:15.752861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:15.752980 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:15.753435 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:15.753444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:15.753447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:15.753710 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:15.753893 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:15.754566 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 10:52:15.754783 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":266,"threads_started":1}
I20260806 10:52:15.756024 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.737508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:36637 (local address 127.16.21.3:58471)
0806 10:52:15.738191 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:15.738233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:15.739927 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:15.740701 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:15.740709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:15.741189 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:15.741713 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.741729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.743121 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.743125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.744658 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.744675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.745774 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.745783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.746060 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.746756 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:15.746779 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:15.749045 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:15.752218 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:15.752227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:15.752241 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:15.752697 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:15.753101 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:15.753106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:15.753110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:15.753302 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:15.753855 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:15.753862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:15.754193 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 10:52:15.754452 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 10:52:15.754956 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":281,"threads_started":1}
I20260806 10:52:15.757436 4046 heartbeater.cc:347] Connected to a master server at 127.16.21.62:36637
I20260806 10:52:15.757745 4046 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:15.758517 4046 heartbeater.cc:511] Master 127.16.21.62:36637 requested a full tablet report, sending...
I20260806 10:52:15.759974 3582 ts_manager.cc:194] Registered new tserver with Master: 17e7399d24574b9db554864df9ad472e (127.16.21.3:33299)
I20260806 10:52:15.760841 3582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.21.3@KRBTEST.COM'} at 127.16.21.3:58471
I20260806 10:52:15.770848 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:52:15.787660 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.775290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.16.21.62:36637)
0806 10:52:15.775452 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:15.775456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:15.776054 (+ 598us) server_negotiation.cc:407] Connection header received
0806 10:52:15.776184 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:15.776188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:15.776301 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:15.776393 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:15.777875 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.778621 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.779862 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.780347 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.780443 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.782882 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:15.782887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:15.782890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:15.782919 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:15.785434 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:15.786033 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:15.786037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:15.786039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:15.786150 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:15.786497 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:15.786500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:15.786502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:15.786801 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:15.786968 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:15.787272 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 10:52:15.787437 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:52:15.791548 3582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:52:15.798439 3582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:52:15.834074 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.809860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33299 (local address 127.0.0.1:49904)
0806 10:52:15.823002 (+ 13142us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:15.823054 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:15.823293 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:15.826077 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:15.826084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:15.826118 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:15.826615 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.826631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.828867 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.828874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.831324 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.831340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.832952 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.832959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.833270 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.833328 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 10:52:15.833467 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 10:52:15.833651 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12809,"mutex_wait_us":62,"thread_start_us":17620,"threads_started":1}
I20260806 10:52:15.834086 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.810328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.16.21.3:33299)
0806 10:52:15.825556 (+ 15228us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:15.825561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:15.825587 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:52:15.825664 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:15.825670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:15.825851 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:15.826060 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:15.826884 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.828702 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.831549 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.832738 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.832880 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.833135 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:15.833537 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 10:52:15.833716 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15100,"thread_start_us":14967,"threads_started":1}
I20260806 10:52:15.835999 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.810842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.16.21.1:37655)
0806 10:52:15.824772 (+ 13930us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:15.824779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:15.827862 (+ 3083us) server_negotiation.cc:407] Connection header received
0806 10:52:15.828173 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:15.828179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:15.828408 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:15.828603 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:15.829201 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.830322 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.833316 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.834779 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.834911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.835134 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:15.835455 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 10:52:15.835635 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13757,"thread_start_us":13597,"threads_started":1}
I20260806 10:52:15.836125 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.810030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:42805 (local address 127.0.0.1:55052)
0806 10:52:15.826823 (+ 16793us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:15.826843 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:15.827195 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:15.828059 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:15.828063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:15.828087 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:15.828486 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.828497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.830395 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.830400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.832949 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.832970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.835421 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.835427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.835597 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.835634 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 10:52:15.835713 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 10:52:15.835823 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16658,"mutex_wait_us":1,"thread_start_us":16494,"threads_started":1}
I20260806 10:52:15.836748 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.810432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37655 (local address 127.0.0.1:35144)
0806 10:52:15.827745 (+ 17313us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:15.827765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:15.827971 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:15.828597 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:15.828601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:15.828629 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:15.829027 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.829038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.830492 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.830503 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.833149 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:15.833161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.835069 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.835075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:15.835235 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.835276 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 10:52:15.836215 (+ 939us) client_negotiation.cc:240] Negotiation successful
0806 10:52:15.836373 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17193,"mutex_wait_us":1,"thread_start_us":13092,"threads_started":1}
I20260806 10:52:15.837469 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:15.810897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.16.21.2:42805)
0806 10:52:15.825146 (+ 14249us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:15.825152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:15.826929 (+ 1777us) server_negotiation.cc:407] Connection header received
0806 10:52:15.827414 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:15.827422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:15.827642 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:15.827787 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:15.828677 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.830231 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.833162 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:15.835219 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:15.835372 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:15.835563 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:15.837011 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0806 10:52:15.837166 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14112,"thread_start_us":14019,"threads_started":1}
I20260806 10:52:15.838893 3980 tablet_service.cc:1467] Processing CreateTablet for tablet 546e232b59704a158650c9b2d5d547ee (DEFAULT_TABLE table=test-table [id=1d5fbe1c2fde49d6bf56f842af566824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:15.839915 3708 tablet_service.cc:1467] Processing CreateTablet for tablet 546e232b59704a158650c9b2d5d547ee (DEFAULT_TABLE table=test-table [id=1d5fbe1c2fde49d6bf56f842af566824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:15.840540 3980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546e232b59704a158650c9b2d5d547ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:15.841185 3708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546e232b59704a158650c9b2d5d547ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:15.841532 3845 tablet_service.cc:1467] Processing CreateTablet for tablet 546e232b59704a158650c9b2d5d547ee (DEFAULT_TABLE table=test-table [id=1d5fbe1c2fde49d6bf56f842af566824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:15.842839 3845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546e232b59704a158650c9b2d5d547ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:15.853248 4066 tablet_bootstrap.cc:492] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Bootstrap starting.
I20260806 10:52:15.856467 4066 tablet_bootstrap.cc:654] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:15.857693 4068 tablet_bootstrap.cc:492] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Bootstrap starting.
I20260806 10:52:15.857939 4066 log.cc:824] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:15.859326 4067 tablet_bootstrap.cc:492] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Bootstrap starting.
I20260806 10:52:15.861482 4068 tablet_bootstrap.cc:654] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:15.862325 4067 tablet_bootstrap.cc:654] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:15.862701 4068 log.cc:824] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:15.862784 4066 tablet_bootstrap.cc:492] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: No bootstrap required, opened a new log
I20260806 10:52:15.863024 4066 ts_tablet_manager.cc:1397] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260806 10:52:15.863374 4067 log.cc:824] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:15.866400 4067 tablet_bootstrap.cc:492] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: No bootstrap required, opened a new log
I20260806 10:52:15.866401 4068 tablet_bootstrap.cc:492] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: No bootstrap required, opened a new log
I20260806 10:52:15.866652 4067 ts_tablet_manager.cc:1397] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 10:52:15.866654 4068 ts_tablet_manager.cc:1397] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 10:52:15.870697 4066 raft_consensus.cc:348] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:15.871059 4066 raft_consensus.cc:374] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:15.871191 4066 raft_consensus.cc:729] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9daf7b7afc41f2a3484a70be691309, State: Initialized, Role: FOLLOWER
I20260806 10:52:15.871805 4066 consensus_queue.cc:260] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:15.873530 4066 ts_tablet_manager.cc:1428] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260806 10:52:15.877452 4067 raft_consensus.cc:348] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:15.877460 4068 raft_consensus.cc:348] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:15.877873 4067 raft_consensus.cc:374] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:15.877874 4068 raft_consensus.cc:374] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:15.877978 4068 raft_consensus.cc:729] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f34131544294f199eb7ca775f977d0d, State: Initialized, Role: FOLLOWER
I20260806 10:52:15.877978 4067 raft_consensus.cc:729] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e7399d24574b9db554864df9ad472e, State: Initialized, Role: FOLLOWER
I20260806 10:52:15.878564 4067 consensus_queue.cc:260] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:15.878564 4068 consensus_queue.cc:260] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:15.879652 3911 heartbeater.cc:503] Master 127.16.21.62:36637 was elected leader, sending a full tablet report...
I20260806 10:52:15.880323 4068 ts_tablet_manager.cc:1428] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 10:52:15.880334 4067 ts_tablet_manager.cc:1428] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260806 10:52:15.880803 4046 heartbeater.cc:503] Master 127.16.21.62:36637 was elected leader, sending a full tablet report...
W20260806 10:52:15.898468 3775 tablet.cc:2369] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:52:15.942983 3912 tablet.cc:2369] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:52:15.986357 4047 tablet.cc:2369] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:16.045306 4073 raft_consensus.cc:482] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:52:16.045630 4073 raft_consensus.cc:504] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:16.047624 4073 leader_election.cc:288] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17e7399d24574b9db554864df9ad472e (127.16.21.3:33299), fc9daf7b7afc41f2a3484a70be691309 (127.16.21.1:37655)
I20260806 10:52:16.054864 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.048108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:43859 (local address 127.16.21.1:37655)
0806 10:52:16.048354 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:16.048361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:16.048770 (+ 409us) server_negotiation.cc:407] Connection header received
0806 10:52:16.049005 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:16.049011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:16.049085 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:16.049187 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:16.049754 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.050708 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.052983 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.053871 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.053974 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.054048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:16.054467 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 10:52:16.054599 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:52:16.054986 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.047953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:37655 (local address 127.16.21.2:43859)
0806 10:52:16.048640 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:16.048655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:16.048843 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:16.049210 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:16.049214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:16.049243 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:16.049606 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:16.049615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.050896 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.050902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:16.052831 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:16.052843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.054101 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.054108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:16.054281 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.054320 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:52:16.054420 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 10:52:16.054527 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":45,"spinlock_wait_cycles":29568,"thread_start_us":721,"threads_started":1}
I20260806 10:52:16.055160 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.048027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:58389 (local address 127.16.21.3:33299)
0806 10:52:16.048243 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:16.048251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:16.049307 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 10:52:16.049520 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:16.049524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:16.049584 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:16.049682 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:16.050344 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.051344 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.053300 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.054147 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.054272 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.054381 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:16.054755 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 10:52:16.054884 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:52:16.055277 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.048447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:33299 (local address 127.16.21.2:58389)
0806 10:52:16.049178 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:16.049192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:16.049358 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:16.049711 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:16.049716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:16.049739 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:16.050195 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:16.050210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.051497 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.051501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:16.053128 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:16.053143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.054354 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.054358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:16.054567 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.054597 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:52:16.054695 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:52:16.054812 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":21,"thread_start_us":462,"threads_started":1}
I20260806 10:52:16.056327 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546e232b59704a158650c9b2d5d547ee" candidate_uuid: "0f34131544294f199eb7ca775f977d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9daf7b7afc41f2a3484a70be691309" is_pre_election: true
I20260806 10:52:16.056444 4001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546e232b59704a158650c9b2d5d547ee" candidate_uuid: "0f34131544294f199eb7ca775f977d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e7399d24574b9db554864df9ad472e" is_pre_election: true
I20260806 10:52:16.056891 3728 raft_consensus.cc:2454] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f34131544294f199eb7ca775f977d0d in term 0.
I20260806 10:52:16.056926 4001 raft_consensus.cc:2454] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f34131544294f199eb7ca775f977d0d in term 0.
I20260806 10:52:16.057587 3799 leader_election.cc:302] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f34131544294f199eb7ca775f977d0d, 17e7399d24574b9db554864df9ad472e, fc9daf7b7afc41f2a3484a70be691309; no voters:
I20260806 10:52:16.058015 4073 raft_consensus.cc:2781] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:52:16.058140 4073 raft_consensus.cc:482] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:52:16.058305 4073 raft_consensus.cc:3037] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:16.060812 4073 raft_consensus.cc:504] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:16.061532 4073 leader_election.cc:288] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [CANDIDATE]: Term 1 election: Requested vote from peers 17e7399d24574b9db554864df9ad472e (127.16.21.3:33299), fc9daf7b7afc41f2a3484a70be691309 (127.16.21.1:37655)
I20260806 10:52:16.061937 4001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546e232b59704a158650c9b2d5d547ee" candidate_uuid: "0f34131544294f199eb7ca775f977d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e7399d24574b9db554864df9ad472e"
I20260806 10:52:16.061936 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546e232b59704a158650c9b2d5d547ee" candidate_uuid: "0f34131544294f199eb7ca775f977d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9daf7b7afc41f2a3484a70be691309"
I20260806 10:52:16.062127 4001 raft_consensus.cc:3037] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:16.062140 3728 raft_consensus.cc:3037] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:16.064349 4001 raft_consensus.cc:2454] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f34131544294f199eb7ca775f977d0d in term 1.
I20260806 10:52:16.064391 3728 raft_consensus.cc:2454] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f34131544294f199eb7ca775f977d0d in term 1.
I20260806 10:52:16.064795 3799 leader_election.cc:302] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f34131544294f199eb7ca775f977d0d, 17e7399d24574b9db554864df9ad472e; no voters:
I20260806 10:52:16.065116 4073 raft_consensus.cc:2781] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:16.065517 4073 raft_consensus.cc:686] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 1 LEADER]: Becoming Leader. State: Replica: 0f34131544294f199eb7ca775f977d0d, State: Running, Role: LEADER
I20260806 10:52:16.066102 4073 consensus_queue.cc:237] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } }
I20260806 10:52:16.070534 3581 catalog_manager.cc:5495] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f34131544294f199eb7ca775f977d0d (127.16.21.2). New cstate: current_term: 1 leader_uuid: "0f34131544294f199eb7ca775f977d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f34131544294f199eb7ca775f977d0d" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 } health_report { overall_health: UNKNOWN } } }
I20260806 10:52:16.129896 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.124844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.16.21.2:42805)
0806 10:52:16.125078 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:16.125082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:16.125100 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:52:16.125228 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:16.125232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:16.125325 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:16.125444 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:52:16.125790 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.126529 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.127831 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.128214 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.128399 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.128834 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:52:16.129311 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:52:16.129417 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:16.129524 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 10:52:16.129587 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 10:52:16.137816 3728 raft_consensus.cc:1261] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 1 FOLLOWER]: Refusing update from remote peer 0f34131544294f199eb7ca775f977d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:16.137832 4001 raft_consensus.cc:1261] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 1 FOLLOWER]: Refusing update from remote peer 0f34131544294f199eb7ca775f977d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:16.138814 4077 consensus_queue.cc:1035] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e7399d24574b9db554864df9ad472e" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:16.139168 4073 consensus_queue.cc:1035] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9daf7b7afc41f2a3484a70be691309" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:16.149968 4080 mvcc.cc:205] Tried to move back new op lower bound from 7315511444009881600 to 7315511443730628608. Current Snapshot: MvccSnapshot[applied={T|T < 7315511444009881600}]
I20260806 10:52:16.152274 4081 mvcc.cc:205] Tried to move back new op lower bound from 7315511444009881600 to 7315511443730628608. Current Snapshot: MvccSnapshot[applied={T|T < 7315511444009881600}]
I20260806 10:52:16.153415 4082 mvcc.cc:205] Tried to move back new op lower bound from 7315511444009881600 to 7315511443730628608. Current Snapshot: MvccSnapshot[applied={T|T < 7315511444009881600}]
I20260806 10:52:16.168237 3582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:16.168615 3582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:16.172384 3582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20c82b8c42604c2cb63113d4aecd1e9b: Sending DeleteTablet for 3 replicas of tablet 546e232b59704a158650c9b2d5d547ee
I20260806 10:52:16.173532 3980 tablet_service.cc:1514] Processing DeleteTablet for tablet 546e232b59704a158650c9b2d5d547ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:16 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:49904
I20260806 10:52:16.173642 3845 tablet_service.cc:1514] Processing DeleteTablet for tablet 546e232b59704a158650c9b2d5d547ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:16 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:55052
I20260806 10:52:16.173970 3708 tablet_service.cc:1514] Processing DeleteTablet for tablet 546e232b59704a158650c9b2d5d547ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:16 UTC) from {username='kudu', principal='kudu/127.16.21.62@KRBTEST.COM'} at 127.0.0.1:35144
I20260806 10:52:16.180683 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3645
I20260806 10:52:16.183940 4091 tablet_replica.cc:330] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: stopping tablet replica
I20260806 10:52:16.184433 4091 raft_consensus.cc:2229] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:16.184952 4091 raft_consensus.cc:2258] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:16.187177 4091 ts_tablet_manager.cc:1905] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:16.188398 4090 tablet_replica.cc:330] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: stopping tablet replica
I20260806 10:52:16.189234 4090 raft_consensus.cc:2229] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:52:16.192711 4090 raft_consensus.cc:2258] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:16.194479 4091 ts_tablet_manager.cc:1918] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:16.194635 4091 log.cc:1196] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-0/wal/wals/546e232b59704a158650c9b2d5d547ee
I20260806 10:52:16.195089 4091 ts_tablet_manager.cc:1939] T 546e232b59704a158650c9b2d5d547ee P fc9daf7b7afc41f2a3484a70be691309: Deleting consensus metadata
I20260806 10:52:16.197054 3566 catalog_manager.cc:4841] TS fc9daf7b7afc41f2a3484a70be691309 (127.16.21.1:37655): tablet 546e232b59704a158650c9b2d5d547ee (table test-table [id=1d5fbe1c2fde49d6bf56f842af566824]) successfully deleted
I20260806 10:52:16.198237 4090 ts_tablet_manager.cc:1905] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 10:52:16 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.1@KRBTEST.COM
I20260806 10:52:16.213189 4089 tablet_replica.cc:330] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: stopping tablet replica
I20260806 10:52:16.213752 4089 raft_consensus.cc:2229] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:16.216320 4090 ts_tablet_manager.cc:1918] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:16.216521 4090 log.cc:1196] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/546e232b59704a158650c9b2d5d547ee
I20260806 10:52:16.217106 4090 ts_tablet_manager.cc:1939] T 546e232b59704a158650c9b2d5d547ee P 0f34131544294f199eb7ca775f977d0d: Deleting consensus metadata
I20260806 10:52:16.216339 4089 raft_consensus.cc:2258] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:16.218961 3568 catalog_manager.cc:4841] TS 0f34131544294f199eb7ca775f977d0d (127.16.21.2:42805): tablet 546e232b59704a158650c9b2d5d547ee (table test-table [id=1d5fbe1c2fde49d6bf56f842af566824]) successfully deleted
I20260806 10:52:16.221062 4089 ts_tablet_manager.cc:1905] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:16.222407 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.181711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.16.21.1:37655)
0806 10:52:16.181899 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:16.181905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:16.183830 (+ 1925us) server_negotiation.cc:407] Connection header received
0806 10:52:16.184949 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:16.184959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:16.185108 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:16.185201 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:16.190337 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.191909 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.198520 (+ 6611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.199439 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.199565 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.214286 (+ 14721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:16.214304 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:16.214320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:16.214366 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:16.218367 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:16.219736 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:16.219745 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:16.219749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:16.219894 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:16.220359 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:16.220364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:16.220367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:16.220764 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:16.221087 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:16.221919 (+ 832us) server_negotiation.cc:299] Negotiation successful
0806 10:52:16.222082 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:52:16.223088 3768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:16.229231 4089 ts_tablet_manager.cc:1918] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:16.229584 4089 log.cc:1196] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/546e232b59704a158650c9b2d5d547ee
I20260806 10:52:16.230463 4089 ts_tablet_manager.cc:1939] T 546e232b59704a158650c9b2d5d547ee P 17e7399d24574b9db554864df9ad472e: Deleting consensus metadata
I20260806 10:52:16.232892 3569 catalog_manager.cc:4841] TS 17e7399d24574b9db554864df9ad472e (127.16.21.3:33299): tablet 546e232b59704a158650c9b2d5d547ee (table test-table [id=1d5fbe1c2fde49d6bf56f842af566824]) successfully deleted
I20260806 10:52:16.334221 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3645
I20260806 10:52:16.355728 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3780
Aug 06 10:52:16 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.2@KRBTEST.COM
I20260806 10:52:16.372834 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.356694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.16.21.2:42805)
0806 10:52:16.356907 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:16.356913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:16.356934 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:52:16.357009 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:16.357018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:16.357116 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:16.357230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:16.358471 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.359794 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.361235 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.361911 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.362039 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.366645 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:16.366659 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:16.366673 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:16.366713 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:16.369866 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:16.370718 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:16.370725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:16.370728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:16.370867 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:16.371259 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:16.371262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:16.371264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:16.371621 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:16.371884 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:16.372186 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 10:52:16.372457 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:52:16.373638 3906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:16.488067 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3780
I20260806 10:52:16.510947 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3917
Aug 06 10:52:16 dist-test-slave-0b58 krb5kdc[3537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.3@KRBTEST.COM
I20260806 10:52:16.525897 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:16.511738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.16.21.3:33299)
0806 10:52:16.511992 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:16.511996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:16.512014 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:52:16.512113 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:16.512117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:16.512177 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:16.512325 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:16.513589 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.514270 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.515476 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:16.515845 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:16.515930 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:16.520542 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:16.520556 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:16.520571 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:16.520604 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:16.523360 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:16.523981 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:16.523988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:16.523990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:16.524127 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:16.524545 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:16.524548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:16.524550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:16.524879 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:16.525092 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:16.525371 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 10:52:16.525523 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 10:52:16.526607 4041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:16.626713 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3917
I20260806 10:52:16.648772 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 3550
I20260806 10:52:16.649896 3613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:16.764981 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 3550
2026-08-06T10:52:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4676 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T10:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:16Z Disabled control of system clock
I20260806 10:52:16.863178 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46367
--webserver_interface=127.16.21.62
--webserver_port=0
--builtin_ntp_servers=127.16.21.20:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:46367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:52:17.203738 4100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:17.203994 4100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:17.204072 4100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:17.211244 4100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:17.211369 4100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:17.211436 4100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:17.211555 4100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:17.211632 4100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:17.220384 4100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:46367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:17.221570 4100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:17.223390 4100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:17.235743 4106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:17.235780 4105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:17.236449 4108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:17.236873 4100 server_base.cc:1034] running on GCE node
I20260806 10:52:17.237681 4100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:17.239133 4100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:17.240373 4100 hybrid_clock.cc:648] HybridClock initialized: now 1786013537240337 us; error 50 us; skew 500 ppm
I20260806 10:52:17.240738 4100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:17.252991 4100 webserver.cc:460] Webserver started at http://127.16.21.62:44819/ using document root <none> and password file <none>
I20260806 10:52:17.253727 4100 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:17.253876 4100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:17.254211 4100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:17.257247 4100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "26aeb30df7874b39ae0563c200a7e801"
format_stamp: "Formatted at 2026-08-06 10:52:17 on dist-test-slave-0b58"
server_key: "c5201452b72614e1925ac75f123c4f4d"
server_key_iv: "b15852e74cbf9adcaff448d9c48c051e"
server_key_version: "encryptionkey@0"
I20260806 10:52:17.258074 4100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "26aeb30df7874b39ae0563c200a7e801"
format_stamp: "Formatted at 2026-08-06 10:52:17 on dist-test-slave-0b58"
server_key: "c5201452b72614e1925ac75f123c4f4d"
server_key_iv: "b15852e74cbf9adcaff448d9c48c051e"
server_key_version: "encryptionkey@0"
I20260806 10:52:17.263144 4100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 10:52:17.266568 4115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:17.267817 4100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:52:17.268002 4100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "26aeb30df7874b39ae0563c200a7e801"
format_stamp: "Formatted at 2026-08-06 10:52:17 on dist-test-slave-0b58"
server_key: "c5201452b72614e1925ac75f123c4f4d"
server_key_iv: "b15852e74cbf9adcaff448d9c48c051e"
server_key_version: "encryptionkey@0"
I20260806 10:52:17.268227 4100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:17.289743 4100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:17.290544 4100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:17.290844 4100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:17.309840 4100 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:46367
I20260806 10:52:17.309878 4167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:46367 every 8 connection(s)
I20260806 10:52:17.311128 4100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:17.312610 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4100
I20260806 10:52:17.312816 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:17.313282 16468 external_mini_cluster.cc:1428] Setting key ef0a3e789d0c3ecbb870ed7538166567
I20260806 10:52:17.315151 4168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:17.326081 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:17.315142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.16.21.62:46367)
0806 10:52:17.316109 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:17.316119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:17.316159 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:52:17.317813 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:17.317832 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:17.318235 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:17.318646 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:17.319189 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:17.320550 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:17.321905 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:17.322540 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:17.322732 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:17.323921 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:17.323928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:17.323943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:17.324536 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:17.324608 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:17.324627 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:17.324750 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:17.324982 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 10:52:17.325252 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":372,"threads_started":1}
I20260806 10:52:17.326437 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801: Bootstrap starting.
I20260806 10:52:17.329708 4168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:17.330621 4168 log.cc:824] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:17.333420 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801: No bootstrap required, opened a new log
I20260806 10:52:17.340631 4168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb30df7874b39ae0563c200a7e801" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46367 } }
I20260806 10:52:17.340960 4168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:17.341032 4168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26aeb30df7874b39ae0563c200a7e801, State: Initialized, Role: FOLLOWER
I20260806 10:52:17.341574 4168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb30df7874b39ae0563c200a7e801" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46367 } }
I20260806 10:52:17.341768 4168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:17.341879 4168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:17.341995 4168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:17.344223 4168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb30df7874b39ae0563c200a7e801" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46367 } }
I20260806 10:52:17.344717 4168 leader_election.cc:302] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26aeb30df7874b39ae0563c200a7e801; no voters:
I20260806 10:52:17.345242 4168 leader_election.cc:288] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:17.345415 4173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:17.346135 4173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [term 1 LEADER]: Becoming Leader. State: Replica: 26aeb30df7874b39ae0563c200a7e801, State: Running, Role: LEADER
I20260806 10:52:17.346658 4173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb30df7874b39ae0563c200a7e801" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46367 } }
I20260806 10:52:17.347223 4168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:17.349696 4175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26aeb30df7874b39ae0563c200a7e801. Latest consensus state: current_term: 1 leader_uuid: "26aeb30df7874b39ae0563c200a7e801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb30df7874b39ae0563c200a7e801" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46367 } } }
I20260806 10:52:17.350302 4175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:17.349756 4174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26aeb30df7874b39ae0563c200a7e801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb30df7874b39ae0563c200a7e801" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46367 } } }
I20260806 10:52:17.350797 4174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:17.352960 4181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:17.358839 4181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:17.366875 4181 catalog_manager.cc:1347] Generated new cluster ID: 24a572099ccb4bf2949bf755c34a1da0
I20260806 10:52:17.366986 4181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:17.385452 4181 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:17.386723 4181 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:17.393847 4181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801: Generated new TSK 0
I20260806 10:52:17.394570 4181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:17.400504 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.1:0
--local_ip_for_outbound_sockets=127.16.21.1
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46367
--builtin_ntp_servers=127.16.21.20:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:52:17.742239 4192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:17.742575 4192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:17.742746 4192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:17.750180 4192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:17.750329 4192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:17.750511 4192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.1
I20260806 10:52:17.759723 4192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.21.1
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.1
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:17.761022 4192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:17.762777 4192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:17.776242 4200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:17.776204 4197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:17.776299 4198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:17.776983 4192 server_base.cc:1034] running on GCE node
I20260806 10:52:17.777627 4192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:17.778566 4192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:17.779779 4192 hybrid_clock.cc:648] HybridClock initialized: now 1786013537779713 us; error 60 us; skew 500 ppm
I20260806 10:52:17.780089 4192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:17.782285 4192 webserver.cc:460] Webserver started at http://127.16.21.1:33369/ using document root <none> and password file <none>
I20260806 10:52:17.782960 4192 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:17.783075 4192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:17.783391 4192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:17.786310 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data/instance:
uuid: "1cdf4f9fe20e4400b4913564e398a627"
format_stamp: "Formatted at 2026-08-06 10:52:17 on dist-test-slave-0b58"
server_key: "c7ec0cb728960d53b3ad7600ccdc4045"
server_key_iv: "bda21710d731d1a46614e28f40b1cb00"
server_key_version: "encryptionkey@0"
I20260806 10:52:17.787402 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance:
uuid: "1cdf4f9fe20e4400b4913564e398a627"
format_stamp: "Formatted at 2026-08-06 10:52:17 on dist-test-slave-0b58"
server_key: "c7ec0cb728960d53b3ad7600ccdc4045"
server_key_iv: "bda21710d731d1a46614e28f40b1cb00"
server_key_version: "encryptionkey@0"
I20260806 10:52:17.794723 4192 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.006s
I20260806 10:52:17.798350 4207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:17.799719 4192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 10:52:17.799901 4192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal
uuid: "1cdf4f9fe20e4400b4913564e398a627"
format_stamp: "Formatted at 2026-08-06 10:52:17 on dist-test-slave-0b58"
server_key: "c7ec0cb728960d53b3ad7600ccdc4045"
server_key_iv: "bda21710d731d1a46614e28f40b1cb00"
server_key_version: "encryptionkey@0"
I20260806 10:52:17.800108 4192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:17.822471 4192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:17.823239 4192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:17.823520 4192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:17.824530 4192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:17.825985 4192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:17.826098 4192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:17.826236 4192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:17.826341 4192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:17.860153 4192 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.1:42157
I20260806 10:52:17.860239 4320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.1:42157 every 8 connection(s)
I20260806 10:52:17.861438 4192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/data/info.pb
I20260806 10:52:17.866329 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4192
I20260806 10:52:17.866499 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-0/wal/instance
I20260806 10:52:17.866940 16468 external_mini_cluster.cc:1428] Setting key edc6269d02bc277999875c2ae6f66a6f
I20260806 10:52:17.870146 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.2:0
--local_ip_for_outbound_sockets=127.16.21.2
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46367
--builtin_ntp_servers=127.16.21.20:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:52:17.874914 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:17.863881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.1:34349 (local address 127.16.21.62:46367)
0806 10:52:17.864358 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:17.864365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:17.865519 (+ 1154us) server_negotiation.cc:407] Connection header received
0806 10:52:17.867545 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:17.867549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:17.867617 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:17.867753 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:17.869096 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:17.870193 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:17.871777 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:17.872554 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:17.872690 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:17.874019 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:17.874028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:17.874035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:17.874059 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:17.874111 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:17.874129 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:17.874224 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:17.874419 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 10:52:17.874522 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260806 10:52:17.875622 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:17.864510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46367 (local address 127.16.21.1:34349)
0806 10:52:17.865368 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:17.865409 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:17.867267 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:17.867976 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:17.867987 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:17.868099 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:17.868917 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:17.868941 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:17.870360 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:17.870367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:17.871599 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:17.871611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:17.872752 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:17.872757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:17.873039 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:17.873711 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:17.873714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:17.873749 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:17.873828 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:17.874226 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:17.874235 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:17.874240 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:52:17.874372 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:52:17.874630 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":259,"threads_started":1}
I20260806 10:52:17.877099 4321 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46367
I20260806 10:52:17.877389 4321 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:17.878052 4321 heartbeater.cc:511] Master 127.16.21.62:46367 requested a full tablet report, sending...
I20260806 10:52:17.879921 4132 ts_manager.cc:194] Registered new tserver with Master: 1cdf4f9fe20e4400b4913564e398a627 (127.16.21.1:42157)
I20260806 10:52:17.881398 4132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.1:34349
W20260806 10:52:18.209723 4325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:18.210029 4325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:18.210220 4325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:18.217602 4325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:18.217733 4325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:18.217921 4325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.2
I20260806 10:52:18.230109 4325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.21.2
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.2
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:18.231458 4325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:18.233296 4325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:18.246014 4332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:18.246054 4331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:18.245915 4334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:18.246922 4325 server_base.cc:1034] running on GCE node
I20260806 10:52:18.247527 4325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:18.248610 4325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:18.249850 4325 hybrid_clock.cc:648] HybridClock initialized: now 1786013538249805 us; error 53 us; skew 500 ppm
I20260806 10:52:18.250186 4325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:18.252539 4325 webserver.cc:460] Webserver started at http://127.16.21.2:35329/ using document root <none> and password file <none>
I20260806 10:52:18.253247 4325 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:18.253387 4325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:18.253718 4325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:18.256784 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data/instance:
uuid: "41daf7d3a4f14df09bf41ed5c017a836"
format_stamp: "Formatted at 2026-08-06 10:52:18 on dist-test-slave-0b58"
server_key: "ba50e5d269d858ef352af7a41ac9d163"
server_key_iv: "f6452259690c1f9aa2435df2d1fb3b13"
server_key_version: "encryptionkey@0"
I20260806 10:52:18.257614 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance:
uuid: "41daf7d3a4f14df09bf41ed5c017a836"
format_stamp: "Formatted at 2026-08-06 10:52:18 on dist-test-slave-0b58"
server_key: "ba50e5d269d858ef352af7a41ac9d163"
server_key_iv: "f6452259690c1f9aa2435df2d1fb3b13"
server_key_version: "encryptionkey@0"
I20260806 10:52:18.262540 4325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:52:18.265924 4340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:18.267114 4325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:52:18.267299 4325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal
uuid: "41daf7d3a4f14df09bf41ed5c017a836"
format_stamp: "Formatted at 2026-08-06 10:52:18 on dist-test-slave-0b58"
server_key: "ba50e5d269d858ef352af7a41ac9d163"
server_key_iv: "f6452259690c1f9aa2435df2d1fb3b13"
server_key_version: "encryptionkey@0"
I20260806 10:52:18.267514 4325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:18.279739 4325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:18.280597 4325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:18.280880 4325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:18.281800 4325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:18.283172 4325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:18.283317 4325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:18.283455 4325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:18.283547 4325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:18.318858 4325 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.2:42789
I20260806 10:52:18.318889 4453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.2:42789 every 8 connection(s)
I20260806 10:52:18.320452 4325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/data/info.pb
I20260806 10:52:18.324335 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4325
I20260806 10:52:18.324512 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal/instance
I20260806 10:52:18.325076 16468 external_mini_cluster.cc:1428] Setting key 907acff843f272c51f00dd8e30e3fb49
I20260806 10:52:18.328641 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.21.3:0
--local_ip_for_outbound_sockets=127.16.21.3
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46367
--builtin_ntp_servers=127.16.21.20:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:52:18.335093 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.323076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.2:42259 (local address 127.16.21.62:46367)
0806 10:52:18.323247 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:18.323251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:18.324966 (+ 1715us) server_negotiation.cc:407] Connection header received
0806 10:52:18.327180 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:18.327185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:18.327277 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:18.327387 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:18.328739 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.329867 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.332335 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.332723 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.332820 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.334425 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:18.334430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:18.334434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:18.334449 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:18.334479 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:18.334490 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:18.334560 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:18.334809 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 10:52:18.334881 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:52:18.336494 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.323641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46367 (local address 127.16.21.2:42259)
0806 10:52:18.324811 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:18.324856 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:18.326828 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:18.327623 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:18.327634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:18.327740 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:18.328526 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.328550 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.330041 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.330050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.332138 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.332155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.332949 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.332958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.333286 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.334078 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:18.334085 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:18.334133 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:18.334225 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:18.334591 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:18.334604 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:18.334614 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:52:18.334791 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 10:52:18.335169 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":359,"threads_started":1}
I20260806 10:52:18.337863 4454 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46367
I20260806 10:52:18.338150 4454 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:18.338851 4454 heartbeater.cc:511] Master 127.16.21.62:46367 requested a full tablet report, sending...
I20260806 10:52:18.340241 4132 ts_manager.cc:194] Registered new tserver with Master: 41daf7d3a4f14df09bf41ed5c017a836 (127.16.21.2:42789)
I20260806 10:52:18.341089 4132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.2:42259
W20260806 10:52:18.668792 4457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:18.669047 4457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:18.669201 4457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:18.676620 4457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:18.676702 4457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:18.676892 4457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.21.3
I20260806 10:52:18.685669 4457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.21.3
--webserver_port=0
--tserver_master_addrs=127.16.21.62:46367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.21.3
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:18.686898 4457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:18.688647 4457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:18.701421 4465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:18.701537 4462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:18.701445 4463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:18.702574 4457 server_base.cc:1034] running on GCE node
I20260806 10:52:18.703198 4457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:18.704104 4457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:18.705377 4457 hybrid_clock.cc:648] HybridClock initialized: now 1786013538705340 us; error 69 us; skew 500 ppm
I20260806 10:52:18.705699 4457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:52:18.707937 4457 webserver.cc:460] Webserver started at http://127.16.21.3:37463/ using document root <none> and password file <none>
I20260806 10:52:18.708688 4457 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:18.708837 4457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:18.709154 4457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:18.712024 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data/instance:
uuid: "c6fd1e9ed602408eb408ea5b9962fa5a"
format_stamp: "Formatted at 2026-08-06 10:52:18 on dist-test-slave-0b58"
server_key: "0804b1044dc520504bfd1462c9dc19dc"
server_key_iv: "07338bc3e90c7514d620b65350a6691b"
server_key_version: "encryptionkey@0"
I20260806 10:52:18.712877 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance:
uuid: "c6fd1e9ed602408eb408ea5b9962fa5a"
format_stamp: "Formatted at 2026-08-06 10:52:18 on dist-test-slave-0b58"
server_key: "0804b1044dc520504bfd1462c9dc19dc"
server_key_iv: "07338bc3e90c7514d620b65350a6691b"
server_key_version: "encryptionkey@0"
I20260806 10:52:18.717654 4457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:52:18.720927 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:18.722132 4457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:52:18.722318 4457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal
uuid: "c6fd1e9ed602408eb408ea5b9962fa5a"
format_stamp: "Formatted at 2026-08-06 10:52:18 on dist-test-slave-0b58"
server_key: "0804b1044dc520504bfd1462c9dc19dc"
server_key_iv: "07338bc3e90c7514d620b65350a6691b"
server_key_version: "encryptionkey@0"
I20260806 10:52:18.722532 4457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:18.738987 4457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:18.739751 4457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:18.740036 4457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:18.741050 4457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:52:18.742394 4457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:52:18.742528 4457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:18.742671 4457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:52:18.742774 4457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:18.777064 4457 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.3:36537
I20260806 10:52:18.777155 4585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.3:36537 every 8 connection(s)
I20260806 10:52:18.778466 4457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/data/info.pb
I20260806 10:52:18.783234 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4457
I20260806 10:52:18.783419 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal/instance
I20260806 10:52:18.783995 16468 external_mini_cluster.cc:1428] Setting key 222e9b2e67ef0a7a61d73e48e3f633f6
I20260806 10:52:18.791200 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.780883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:51001 (local address 127.16.21.62:46367)
0806 10:52:18.781070 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:18.781075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:18.782383 (+ 1308us) server_negotiation.cc:407] Connection header received
0806 10:52:18.784839 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:18.784844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:18.784917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:18.785033 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:18.786359 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.787382 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.788902 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.789305 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.789397 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.790570 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:18.790575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:18.790578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:18.790595 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:18.790627 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:18.790638 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:18.790709 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:18.790905 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 10:52:18.790976 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:52:18.792054 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.781390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:46367 (local address 127.16.21.3:51001)
0806 10:52:18.782231 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:18.782273 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:18.784555 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:18.785241 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:18.785253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:18.785370 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:18.786186 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.786204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.787514 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.787518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.788781 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.788793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.789480 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.789484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.789669 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.790291 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:52:18.790294 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:18.790327 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:52:18.790399 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:18.790722 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:18.790730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:18.790735 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:52:18.790862 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 10:52:18.791121 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":262,"threads_started":1}
I20260806 10:52:18.793372 4586 heartbeater.cc:347] Connected to a master server at 127.16.21.62:46367
I20260806 10:52:18.793661 4586 heartbeater.cc:464] Registering TS with master...
I20260806 10:52:18.794353 4586 heartbeater.cc:511] Master 127.16.21.62:46367 requested a full tablet report, sending...
I20260806 10:52:18.795601 4132 ts_manager.cc:194] Registered new tserver with Master: c6fd1e9ed602408eb408ea5b9962fa5a (127.16.21.3:36537)
I20260806 10:52:18.796446 4132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.21.3:51001
I20260806 10:52:18.799297 16468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:52:18.809122 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.803513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.16.21.62:46367)
0806 10:52:18.803802 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:18.803807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:18.804100 (+ 293us) server_negotiation.cc:407] Connection header received
0806 10:52:18.804249 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:18.804298 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:18.804385 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:18.804468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:18.804965 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.805641 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.806962 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.807357 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.807452 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.808453 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:18.808458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:18.808461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:18.808477 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:18.808507 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:18.808518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:18.808591 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:18.808719 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 10:52:18.808795 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:52:18.813372 4132 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:52:18.820181 4132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:52:18.842500 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.831978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.3:36537 (local address 127.0.0.1:60372)
0806 10:52:18.833645 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:18.833668 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:18.833819 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:18.834809 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:18.834820 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:18.834862 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:18.835313 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.835327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.837166 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.837171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.839635 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.839650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.841505 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.841512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.841720 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.841774 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 10:52:18.841915 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 10:52:18.842099 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"mutex_wait_us":23,"thread_start_us":529,"threads_started":1}
I20260806 10:52:18.843012 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.831363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43794)
0806 10:52:18.833000 (+ 1637us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:18.833044 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:18.833235 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:18.835615 (+ 2380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:18.835619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:18.835647 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:18.836146 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.836163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.837923 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.837931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.840290 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.840306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.842013 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.842022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.842390 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.842429 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:52:18.842505 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:52:18.842634 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"mutex_wait_us":274,"thread_start_us":1915,"threads_started":1}
I20260806 10:52:18.843560 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.833663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:42789 (local address 127.0.0.1:58882)
0806 10:52:18.834607 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:18.834626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:18.834740 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:18.835841 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:18.835846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:18.835873 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:18.836381 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.836400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.838310 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.838315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.840375 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:18.840388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.842155 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.842161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:18.842323 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.842362 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:52:18.842427 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 10:52:18.842602 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":72,"thread_start_us":774,"threads_started":1}
I20260806 10:52:18.843874 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.832149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.16.21.3:36537)
0806 10:52:18.833864 (+ 1715us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:18.833874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:18.833911 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:52:18.834005 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:18.834014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:18.834263 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:18.834454 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:18.835544 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.837018 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.839856 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.841298 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.841431 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.841699 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:18.843373 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0806 10:52:18.843519 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":2130,"threads_started":1}
I20260806 10:52:18.843864 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.834146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.16.21.2:42789)
0806 10:52:18.835214 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:18.835222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:18.835250 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:52:18.835358 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:18.835366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:18.835616 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:18.836540 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:18.836744 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.838116 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.840552 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.841943 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.842993 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.843257 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:18.843328 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 10:52:18.843499 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":785,"threads_started":1}
I20260806 10:52:18.843928 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:18.831510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.16.21.1:42157)
0806 10:52:18.834320 (+ 2810us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:18.834331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:18.834363 (+ 32us) server_negotiation.cc:407] Connection header received
0806 10:52:18.835141 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:18.835163 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:18.835412 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:18.835605 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:18.836369 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.837795 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.840500 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:18.841802 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:18.841940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:18.842158 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:18.843328 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0806 10:52:18.843512 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2651,"thread_start_us":2553,"threads_started":1}
I20260806 10:52:18.846833 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 88410198e5cb436a8f02a401bbff939b (DEFAULT_TABLE table=test-table [id=21c833fffc5d46ecbabb66c00da91916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:18.847402 4519 tablet_service.cc:1467] Processing CreateTablet for tablet 88410198e5cb436a8f02a401bbff939b (DEFAULT_TABLE table=test-table [id=21c833fffc5d46ecbabb66c00da91916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:18.847384 4255 tablet_service.cc:1467] Processing CreateTablet for tablet 88410198e5cb436a8f02a401bbff939b (DEFAULT_TABLE table=test-table [id=21c833fffc5d46ecbabb66c00da91916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:52:18.848300 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88410198e5cb436a8f02a401bbff939b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:18.848701 4255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88410198e5cb436a8f02a401bbff939b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:18.848702 4519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88410198e5cb436a8f02a401bbff939b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:18.859510 4604 tablet_bootstrap.cc:492] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Bootstrap starting.
I20260806 10:52:18.863932 4604 tablet_bootstrap.cc:654] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:18.865298 4604 log.cc:824] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:18.875600 4606 tablet_bootstrap.cc:492] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Bootstrap starting.
I20260806 10:52:18.876431 4605 tablet_bootstrap.cc:492] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Bootstrap starting.
I20260806 10:52:18.877401 4604 tablet_bootstrap.cc:492] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: No bootstrap required, opened a new log
I20260806 10:52:18.877676 4604 ts_tablet_manager.cc:1397] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260806 10:52:18.880012 4605 tablet_bootstrap.cc:654] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:18.881285 4605 log.cc:824] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:18.883132 4606 tablet_bootstrap.cc:654] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:18.884029 4321 heartbeater.cc:503] Master 127.16.21.62:46367 was elected leader, sending a full tablet report...
I20260806 10:52:18.884306 4605 tablet_bootstrap.cc:492] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: No bootstrap required, opened a new log
I20260806 10:52:18.884264 4606 log.cc:824] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:18.884562 4605 ts_tablet_manager.cc:1397] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260806 10:52:18.887823 4604 raft_consensus.cc:348] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:18.888242 4604 raft_consensus.cc:374] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:18.888408 4604 raft_consensus.cc:729] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41daf7d3a4f14df09bf41ed5c017a836, State: Initialized, Role: FOLLOWER
I20260806 10:52:18.889117 4604 consensus_queue.cc:260] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:18.891897 4606 tablet_bootstrap.cc:492] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: No bootstrap required, opened a new log
I20260806 10:52:18.892213 4606 ts_tablet_manager.cc:1397] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260806 10:52:18.896314 4454 heartbeater.cc:503] Master 127.16.21.62:46367 was elected leader, sending a full tablet report...
I20260806 10:52:18.896778 4605 raft_consensus.cc:348] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:18.897210 4605 raft_consensus.cc:374] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:18.897336 4604 ts_tablet_manager.cc:1428] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260806 10:52:18.897311 4605 raft_consensus.cc:729] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdf4f9fe20e4400b4913564e398a627, State: Initialized, Role: FOLLOWER
I20260806 10:52:18.898026 4605 consensus_queue.cc:260] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:18.899869 4605 ts_tablet_manager.cc:1428] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260806 10:52:18.902046 4606 raft_consensus.cc:348] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:18.902523 4606 raft_consensus.cc:374] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:18.902643 4606 raft_consensus.cc:729] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fd1e9ed602408eb408ea5b9962fa5a, State: Initialized, Role: FOLLOWER
I20260806 10:52:18.903261 4606 consensus_queue.cc:260] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:18.904336 4586 heartbeater.cc:503] Master 127.16.21.62:46367 was elected leader, sending a full tablet report...
I20260806 10:52:18.904789 4606 ts_tablet_manager.cc:1428] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260806 10:52:19.030195 4587 tablet.cc:2369] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:19.046056 4612 raft_consensus.cc:482] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:52:19.046427 4612 raft_consensus.cc:504] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:19.048122 4612 leader_election.cc:288] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cdf4f9fe20e4400b4913564e398a627 (127.16.21.1:42157), 41daf7d3a4f14df09bf41ed5c017a836 (127.16.21.2:42789)
I20260806 10:52:19.054173 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.048538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.16.21.3:45973)
0806 10:52:19.048735 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.048755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:19.048879 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:19.049302 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:19.049307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:19.049341 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:19.049593 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:19.049605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.050621 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.050628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:19.052446 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:19.052460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.053538 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.053544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:19.053764 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.053789 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 10:52:19.053867 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 10:52:19.053959 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":22}
I20260806 10:52:19.054340 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.048651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:45973 (local address 127.16.21.1:42157)
0806 10:52:19.048858 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:19.048862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:19.048881 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:52:19.049022 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:19.049028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:19.049117 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:19.049219 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:19.049769 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.050477 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.052601 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.053387 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.053482 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.053546 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:19.053925 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 10:52:19.054057 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:52:19.055428 4275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88410198e5cb436a8f02a401bbff939b" candidate_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdf4f9fe20e4400b4913564e398a627" is_pre_election: true
I20260806 10:52:19.055810 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.049119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.21.3:38295 (local address 127.16.21.2:42789)
0806 10:52:19.049383 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:19.049392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:19.049894 (+ 502us) server_negotiation.cc:407] Connection header received
0806 10:52:19.050075 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:19.050079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:19.050151 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:19.050248 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:52:19.050892 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.051568 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.053740 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.054915 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.055005 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.055064 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:19.055506 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 10:52:19.055593 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:52:19.055968 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.048981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.2:42789 (local address 127.16.21.3:38295)
0806 10:52:19.049794 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.049808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:19.049937 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:19.050264 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:19.050268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:19.050292 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:52:19.050710 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:19.050727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.051721 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.051727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:19.053551 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:19.053563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.055051 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.055059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:19.055315 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.055359 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 10:52:19.055449 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 10:52:19.055575 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":6,"thread_start_us":471,"threads_started":1}
I20260806 10:52:19.056018 4275 raft_consensus.cc:2454] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6fd1e9ed602408eb408ea5b9962fa5a in term 0.
I20260806 10:52:19.056746 4475 leader_election.cc:302] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdf4f9fe20e4400b4913564e398a627, c6fd1e9ed602408eb408ea5b9962fa5a; no voters:
I20260806 10:52:19.057080 4612 raft_consensus.cc:2781] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:52:19.057015 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88410198e5cb436a8f02a401bbff939b" candidate_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41daf7d3a4f14df09bf41ed5c017a836" is_pre_election: true
I20260806 10:52:19.057224 4612 raft_consensus.cc:482] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:52:19.057322 4612 raft_consensus.cc:3037] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:19.057528 4408 raft_consensus.cc:2454] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6fd1e9ed602408eb408ea5b9962fa5a in term 0.
I20260806 10:52:19.059677 4612 raft_consensus.cc:504] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:19.060328 4612 leader_election.cc:288] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [CANDIDATE]: Term 1 election: Requested vote from peers 1cdf4f9fe20e4400b4913564e398a627 (127.16.21.1:42157), 41daf7d3a4f14df09bf41ed5c017a836 (127.16.21.2:42789)
I20260806 10:52:19.060722 4275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88410198e5cb436a8f02a401bbff939b" candidate_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdf4f9fe20e4400b4913564e398a627"
I20260806 10:52:19.060822 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88410198e5cb436a8f02a401bbff939b" candidate_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41daf7d3a4f14df09bf41ed5c017a836"
I20260806 10:52:19.060940 4275 raft_consensus.cc:3037] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:19.060987 4408 raft_consensus.cc:3037] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:19.064162 4275 raft_consensus.cc:2454] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6fd1e9ed602408eb408ea5b9962fa5a in term 1.
I20260806 10:52:19.064169 4408 raft_consensus.cc:2454] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6fd1e9ed602408eb408ea5b9962fa5a in term 1.
I20260806 10:52:19.064653 4475 leader_election.cc:302] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41daf7d3a4f14df09bf41ed5c017a836, c6fd1e9ed602408eb408ea5b9962fa5a; no voters:
I20260806 10:52:19.064975 4612 raft_consensus.cc:2781] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:19.065413 4612 raft_consensus.cc:686] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 1 LEADER]: Becoming Leader. State: Replica: c6fd1e9ed602408eb408ea5b9962fa5a, State: Running, Role: LEADER
I20260806 10:52:19.065955 4612 consensus_queue.cc:237] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } }
I20260806 10:52:19.070343 4131 catalog_manager.cc:5495] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a reported cstate change: term changed from 0 to 1, leader changed from <none> to c6fd1e9ed602408eb408ea5b9962fa5a (127.16.21.3). New cstate: current_term: 1 leader_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6fd1e9ed602408eb408ea5b9962fa5a" member_type: VOTER last_known_addr { host: "127.16.21.3" port: 36537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 } health_report { overall_health: UNKNOWN } } }
W20260806 10:52:19.072644 4455 tablet.cc:2369] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:52:19.113886 4322 tablet.cc:2369] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:19.133539 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.127955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.16.21.3:36537)
0806 10:52:19.128171 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:19.128178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:19.128322 (+ 144us) server_negotiation.cc:407] Connection header received
0806 10:52:19.128577 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:19.128581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:19.128666 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:19.128823 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:52:19.129224 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.129927 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.131438 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.131831 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.131963 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.132503 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:52:19.132971 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:52:19.133127 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:19.133210 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 10:52:19.133292 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:52:19.141227 4275 raft_consensus.cc:1261] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 1 FOLLOWER]: Refusing update from remote peer c6fd1e9ed602408eb408ea5b9962fa5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:19.141227 4408 raft_consensus.cc:1261] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 1 FOLLOWER]: Refusing update from remote peer c6fd1e9ed602408eb408ea5b9962fa5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:19.142191 4614 consensus_queue.cc:1035] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41daf7d3a4f14df09bf41ed5c017a836" member_type: VOTER last_known_addr { host: "127.16.21.2" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:19.142568 4612 consensus_queue.cc:1035] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cdf4f9fe20e4400b4913564e398a627" member_type: VOTER last_known_addr { host: "127.16.21.1" port: 42157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:19.170210 4132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:19.170499 4132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:52:19.174381 4132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26aeb30df7874b39ae0563c200a7e801: Sending DeleteTablet for 3 replicas of tablet 88410198e5cb436a8f02a401bbff939b
I20260806 10:52:19.175490 4519 tablet_service.cc:1514] Processing DeleteTablet for tablet 88410198e5cb436a8f02a401bbff939b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:19 UTC) from {username='slave'} at 127.0.0.1:60372
I20260806 10:52:19.175647 4255 tablet_service.cc:1514] Processing DeleteTablet for tablet 88410198e5cb436a8f02a401bbff939b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:19 UTC) from {username='slave'} at 127.0.0.1:43794
I20260806 10:52:19.175951 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet 88410198e5cb436a8f02a401bbff939b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:52:19 UTC) from {username='slave'} at 127.0.0.1:58882
I20260806 10:52:19.176852 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4192
I20260806 10:52:19.190987 4627 tablet_replica.cc:330] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: stopping tablet replica
I20260806 10:52:19.191785 4627 raft_consensus.cc:2229] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:52:19.192736 4627 raft_consensus.cc:2258] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:19.195297 4627 ts_tablet_manager.cc:1905] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:19.196554 4629 tablet_replica.cc:330] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: stopping tablet replica
I20260806 10:52:19.196794 4628 tablet_replica.cc:330] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: stopping tablet replica
I20260806 10:52:19.197285 4628 raft_consensus.cc:2229] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:19.197855 4628 raft_consensus.cc:2258] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:19.198819 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.178607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.16.21.1:42157)
0806 10:52:19.178822 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:19.178831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:19.189041 (+ 10210us) server_negotiation.cc:407] Connection header received
0806 10:52:19.189501 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:19.189507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:19.189620 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:19.189712 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:19.190391 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.191561 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.193830 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.194800 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.194930 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.197071 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:19.197082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:19.197099 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:19.197882 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:19.197942 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:19.197969 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:19.198054 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:19.198331 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 10:52:19.198465 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:52:19.200474 4316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:19.203043 4629 raft_consensus.cc:2229] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:52:19.203588 4629 raft_consensus.cc:2258] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:52:19.206252 4629 ts_tablet_manager.cc:1905] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:19.208086 4627 ts_tablet_manager.cc:1918] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:19.208271 4627 log.cc:1196] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-2/wal/wals/88410198e5cb436a8f02a401bbff939b
I20260806 10:52:19.208956 4627 ts_tablet_manager.cc:1939] T 88410198e5cb436a8f02a401bbff939b P c6fd1e9ed602408eb408ea5b9962fa5a: Deleting consensus metadata
I20260806 10:52:19.210894 4117 catalog_manager.cc:4841] TS c6fd1e9ed602408eb408ea5b9962fa5a (127.16.21.3:36537): tablet 88410198e5cb436a8f02a401bbff939b (table test-table [id=21c833fffc5d46ecbabb66c00da91916]) successfully deleted
I20260806 10:52:19.212827 4629 ts_tablet_manager.cc:1918] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:52:19.212993 4629 log.cc:1196] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Deleting WAL directory at /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013221924036-16468-0/minicluster-data/ts-1/wal/wals/88410198e5cb436a8f02a401bbff939b
I20260806 10:52:19.213522 4629 ts_tablet_manager.cc:1939] T 88410198e5cb436a8f02a401bbff939b P 41daf7d3a4f14df09bf41ed5c017a836: Deleting consensus metadata
I20260806 10:52:19.214977 4119 catalog_manager.cc:4841] TS 41daf7d3a4f14df09bf41ed5c017a836 (127.16.21.2:42789): tablet 88410198e5cb436a8f02a401bbff939b (table test-table [id=21c833fffc5d46ecbabb66c00da91916]) successfully deleted
I20260806 10:52:19.200591 4628 ts_tablet_manager.cc:1905] T 88410198e5cb436a8f02a401bbff939b P 1cdf4f9fe20e4400b4913564e398a627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:52:19.356416 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4192
I20260806 10:52:19.377653 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4325
W20260806 10:52:19.377650 4119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.21.1:42157 (error 108)
I20260806 10:52:19.379798 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.379016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43814)
0806 10:52:19.379462 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.379598 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
W20260806 10:52:19.380190 4119 catalog_manager.cc:4584] TS 1cdf4f9fe20e4400b4913564e398a627 (127.16.21.1:42157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88410198e5cb436a8f02a401bbff939b: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
I20260806 10:52:19.385234 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.378727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.16.21.2:42789)
0806 10:52:19.378925 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:19.378931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:19.378950 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:52:19.379017 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:19.379023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:19.379144 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:19.379280 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:19.379821 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.380828 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.382135 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.382543 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.382675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.383806 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:19.383815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:19.383827 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:19.384461 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:19.384507 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:19.384527 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:19.384657 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:19.384786 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 10:52:19.384889 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:52:19.385984 4449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:19.433789 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.433235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43820)
0806 10:52:19.433496 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.433619 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 10:52:19.435151 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.434718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43822)
0806 10:52:19.434910 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.434994 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 10:52:19.475438 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4325
I20260806 10:52:19.494972 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4457
I20260806 10:52:19.495867 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.495353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43836)
0806 10:52:19.495568 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.495687 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 10:52:19.497135 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.496735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43850)
0806 10:52:19.496902 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.496973 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 10:52:19.503198 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.495813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.16.21.3:36537)
0806 10:52:19.496106 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:19.496112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:19.496136 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:52:19.496385 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:19.496391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:19.496479 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:19.496632 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:19.497106 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.498047 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.499917 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:19.500443 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:19.500546 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:19.501696 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:19.501703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:19.501717 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:19.502320 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:19.502367 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:19.502392 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:19.502467 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:19.502664 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 10:52:19.502737 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 10:52:19.503986 4581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:19.578004 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.577509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43852)
0806 10:52:19.577747 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.577842 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 10:52:19.579131 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:19.578728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.1:42157 (local address 127.0.0.1:43856)
0806 10:52:19.578940 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:19.579004 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 10:52:19.625576 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4457
I20260806 10:52:19.648382 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4100
I20260806 10:52:19.649595 4163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:19.760757 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4100
2026-08-06T10:52:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2994 ms)
[----------] 6 tests from EncryptionPolicyTest (22096 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:19 dist-test-slave-0b58 krb5kdc[4639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:19 dist-test-slave-0b58 krb5kdc[4639](info): set up 2 sockets
Aug 06 10:52:19 dist-test-slave-0b58 krb5kdc[4639](info): commencing operation
krb5kdc: starting...
W20260806 10:52:20.896236 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:21 dist-test-slave-0b58 krb5kdc[4639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:21.208343 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:21.542255 4653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:21.542589 4653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:21.542768 4653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:21.549679 4653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:21.549830 4653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:21.549932 4653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:21.550050 4653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:21.550136 4653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:21.559530 4653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:21.560813 4653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:21.562177 4653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:21.574010 4658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:21.574069 4659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:21.573972 4661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:21.575047 4653 server_base.cc:1034] running on GCE node
I20260806 10:52:21.575803 4653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:21.577304 4653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:21.578544 4653 hybrid_clock.cc:648] HybridClock initialized: now 1786013541578508 us; error 50 us; skew 500 ppm
I20260806 10:52:21.578902 4653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:21 dist-test-slave-0b58 krb5kdc[4639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:21.582052 4653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:21.583525 4653 webserver.cc:460] Webserver started at http://127.0.0.1:39279/ using document root <none> and password file <none>
I20260806 10:52:21.584146 4653 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:21.584246 4653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:21.584578 4653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:21.587477 4653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "c436b5173e584e72bb76a50483f7de77"
format_stamp: "Formatted at 2026-08-06 10:52:21 on dist-test-slave-0b58"
server_key: "3e7a92a1c001ed547d929e340ecc0cec"
server_key_iv: "4eca1cba9b0ab29d738363bd17272c69"
server_key_version: "encryptionkey@0"
I20260806 10:52:21.588224 4653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "c436b5173e584e72bb76a50483f7de77"
format_stamp: "Formatted at 2026-08-06 10:52:21 on dist-test-slave-0b58"
server_key: "3e7a92a1c001ed547d929e340ecc0cec"
server_key_iv: "4eca1cba9b0ab29d738363bd17272c69"
server_key_version: "encryptionkey@0"
I20260806 10:52:21.593293 4653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 10:52:21.596653 4668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:21.597896 4653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:52:21.598071 4653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "c436b5173e584e72bb76a50483f7de77"
format_stamp: "Formatted at 2026-08-06 10:52:21 on dist-test-slave-0b58"
server_key: "3e7a92a1c001ed547d929e340ecc0cec"
server_key_iv: "4eca1cba9b0ab29d738363bd17272c69"
server_key_version: "encryptionkey@0"
I20260806 10:52:21.598243 4653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:21.629001 4653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:21.633068 4653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:21.633469 4653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:21.652771 4653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44125
I20260806 10:52:21.652820 4719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44125 every 8 connection(s)
I20260806 10:52:21.654107 4653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:21.657786 4720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:21.659870 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4653
I20260806 10:52:21.660112 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:21.660965 16468 external_mini_cluster.cc:1428] Setting key 1450b88bea2bc77e57b8b41e24e626c6
I20260806 10:52:21.670570 4720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77: Bootstrap starting.
I20260806 10:52:21.674376 4720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:21.675658 4720 log.cc:824] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:52:21 dist-test-slave-0b58 krb5kdc[4639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:52:21.679051 4720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77: No bootstrap required, opened a new log
I20260806 10:52:21.685815 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:21.662959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35045 (local address 127.0.0.1:44125)
0806 10:52:21.663782 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:21.663796 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:21.663837 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:52:21.665549 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:21.665570 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:21.665917 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:21.666233 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:21.668131 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:21.669511 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:21.671304 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:21.671933 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:21.672066 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:21.677388 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:21.677398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:21.677415 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:21.677461 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:21.681400 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:21.682154 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:21.682163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:21.682176 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:21.682407 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:21.682826 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:21.682832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:21.682835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:21.683496 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:21.683822 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:21.684134 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 10:52:21.684532 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":378,"threads_started":1}
I20260806 10:52:21.689695 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:21.689707 4720 raft_consensus.cc:348] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c436b5173e584e72bb76a50483f7de77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260806 10:52:21.690095 4720 raft_consensus.cc:374] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:21.690208 4720 raft_consensus.cc:729] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c436b5173e584e72bb76a50483f7de77, State: Initialized, Role: FOLLOWER
I20260806 10:52:21.690711 4720 consensus_queue.cc:260] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c436b5173e584e72bb76a50483f7de77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260806 10:52:21.690953 4720 raft_consensus.cc:388] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:21.691103 4720 raft_consensus.cc:482] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:21.691277 4720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:21.693420 4720 raft_consensus.cc:504] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c436b5173e584e72bb76a50483f7de77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260806 10:52:21.693897 4720 leader_election.cc:302] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c436b5173e584e72bb76a50483f7de77; no voters:
I20260806 10:52:21.694470 4720 leader_election.cc:288] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:21.694624 4729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:21.696058 4729 raft_consensus.cc:686] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [term 1 LEADER]: Becoming Leader. State: Replica: c436b5173e584e72bb76a50483f7de77, State: Running, Role: LEADER
I20260806 10:52:21.696693 4729 consensus_queue.cc:237] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c436b5173e584e72bb76a50483f7de77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260806 10:52:21.696875 4720 sys_catalog.cc:564] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:21.702411 4730 sys_catalog.cc:455] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c436b5173e584e72bb76a50483f7de77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c436b5173e584e72bb76a50483f7de77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } } }
I20260806 10:52:21.702981 4731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c436b5173e584e72bb76a50483f7de77. Latest consensus state: current_term: 1 leader_uuid: "c436b5173e584e72bb76a50483f7de77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c436b5173e584e72bb76a50483f7de77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } } }
I20260806 10:52:21.703296 4730 sys_catalog.cc:458] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:21.703362 4731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77 [sys.catalog]: This master's current role is: LEADER
W20260806 10:52:21.717235 4746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:52:21.717617 4746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:52:21.717825 4737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:21.718473 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:21.695535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52287 (local address 127.0.0.1:44125)
0806 10:52:21.696400 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:21.696406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:21.697059 (+ 653us) server_negotiation.cc:407] Connection header received
0806 10:52:21.697258 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:21.697263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:21.697334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:21.697455 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:21.699677 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:21.700714 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:21.703143 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:21.704343 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:21.704498 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:21.709356 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:21.709369 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:21.709377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:21.709431 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:21.714320 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:21.715078 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:21.715084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:21.715087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:21.715336 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:21.715982 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:21.715987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:21.715990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:21.716455 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:21.717703 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:21.717890 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 10:52:21.718077 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759}
I20260806 10:52:21.719933 4737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:21.727850 4737 catalog_manager.cc:1347] Generated new cluster ID: 0c486b59b6e247aab1093a5c77397ce4
I20260806 10:52:21.727949 4737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:21.743220 4737 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:21.744968 4737 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:21.755393 4737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c436b5173e584e72bb76a50483f7de77: Generated new TSK 0
I20260806 10:52:21.756328 4737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:21.768759 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4653
I20260806 10:52:21.770882 4715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:21.862140 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4653
W20260806 10:52:21.880342 4637 connection.cc:535] client connection to 127.0.0.1:44125 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44125): Connection reset by peer (error 104)
2026-08-06T10:52:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2092 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:21 dist-test-slave-0b58 krb5kdc[4752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:21 dist-test-slave-0b58 krb5kdc[4752](info): set up 2 sockets
Aug 06 10:52:21 dist-test-slave-0b58 krb5kdc[4752](info): commencing operation
krb5kdc: starting...
W20260806 10:52:23.008249 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:23 dist-test-slave-0b58 krb5kdc[4752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:23.321151 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:23.653447 4765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:23.653754 4765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:23.653929 4765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:23.661499 4765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:23.661630 4765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:23.661741 4765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:23.661851 4765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:23.661928 4765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:23.672029 4765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:23.673453 4765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:23.674795 4765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:23.686345 4770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:23.686475 4771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:23.686587 4773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:23.687536 4765 server_base.cc:1034] running on GCE node
I20260806 10:52:23.688319 4765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:23.689656 4765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:23.690856 4765 hybrid_clock.cc:648] HybridClock initialized: now 1786013543690821 us; error 33 us; skew 500 ppm
I20260806 10:52:23.691197 4765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:23 dist-test-slave-0b58 krb5kdc[4752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:23.699565 4765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:23.701161 4765 webserver.cc:460] Webserver started at http://127.0.0.1:45637/ using document root <none> and password file <none>
I20260806 10:52:23.701840 4765 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:23.701993 4765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:23.702307 4765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:23.705380 4765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "5da3ab5fddc2401eb19fd9608559bc8c"
format_stamp: "Formatted at 2026-08-06 10:52:23 on dist-test-slave-0b58"
server_key: "2cc71476190c1551625019813fa685af"
server_key_iv: "b5fcd7b6f975955a6573cdfaa9c67acf"
server_key_version: "encryptionkey@0"
I20260806 10:52:23.706182 4765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "5da3ab5fddc2401eb19fd9608559bc8c"
format_stamp: "Formatted at 2026-08-06 10:52:23 on dist-test-slave-0b58"
server_key: "2cc71476190c1551625019813fa685af"
server_key_iv: "b5fcd7b6f975955a6573cdfaa9c67acf"
server_key_version: "encryptionkey@0"
I20260806 10:52:23.711277 4765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 10:52:23.714780 4782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:23.715983 4765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:52:23.716162 4765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "5da3ab5fddc2401eb19fd9608559bc8c"
format_stamp: "Formatted at 2026-08-06 10:52:23 on dist-test-slave-0b58"
server_key: "2cc71476190c1551625019813fa685af"
server_key_iv: "b5fcd7b6f975955a6573cdfaa9c67acf"
server_key_version: "encryptionkey@0"
I20260806 10:52:23.716416 4765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:23.746785 4765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:23.751616 4765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:23.751917 4765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:23.771631 4765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33011
I20260806 10:52:23.771667 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33011 every 8 connection(s)
I20260806 10:52:23.773012 4765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:23.776607 4834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:23.780697 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4765
I20260806 10:52:23.780901 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:23.781349 16468 external_mini_cluster.cc:1428] Setting key 06ed3e5c33263f7b487a33ab158caf85
I20260806 10:52:23.787456 4834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c: Bootstrap starting.
I20260806 10:52:23.790980 4834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:23.792230 4834 log.cc:824] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:23.795153 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:23.783156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49979 (local address 127.0.0.1:33011)
0806 10:52:23.783981 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:23.783992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:23.784037 (+ 45us) server_negotiation.cc:407] Connection header received
0806 10:52:23.786015 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:23.786036 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:23.786408 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:23.786698 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:23.787322 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:23.788718 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:23.790587 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:23.791299 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:23.791426 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:23.792711 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:23.792718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:23.792753 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:23.793534 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:23.793626 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:23.793665 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:23.793782 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:23.794059 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 10:52:23.794252 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":404,"threads_started":1}
I20260806 10:52:23.795240 4834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c: No bootstrap required, opened a new log
I20260806 10:52:23.798872 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:23.805181 4834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260806 10:52:23.805657 4834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:23.805791 4834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da3ab5fddc2401eb19fd9608559bc8c, State: Initialized, Role: FOLLOWER
I20260806 10:52:23.806452 4834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260806 10:52:23.806735 4834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:23.806879 4834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:23.807073 4834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:23.809789 4834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260806 10:52:23.810346 4834 leader_election.cc:302] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5da3ab5fddc2401eb19fd9608559bc8c; no voters:
I20260806 10:52:23.811005 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:23.804032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49441 (local address 127.0.0.1:33011)
0806 10:52:23.804242 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:23.804251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:23.804743 (+ 492us) server_negotiation.cc:407] Connection header received
0806 10:52:23.804937 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:23.804941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:23.804997 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:23.805067 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:23.805731 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:23.806881 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:23.808228 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:23.808969 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:23.809092 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:23.810232 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:23.810238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:23.810243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:23.810270 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:23.810315 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:23.810344 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:23.810462 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:23.810683 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 10:52:23.810736 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:52:23.812569 4834 leader_election.cc:288] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:23.812701 4844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:23.813556 4844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [term 1 LEADER]: Becoming Leader. State: Replica: 5da3ab5fddc2401eb19fd9608559bc8c, State: Running, Role: LEADER
I20260806 10:52:23.814127 4844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260806 10:52:23.814841 4834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:23.817384 4846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5da3ab5fddc2401eb19fd9608559bc8c. Latest consensus state: current_term: 1 leader_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } } }
I20260806 10:52:23.817356 4845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da3ab5fddc2401eb19fd9608559bc8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } } }
I20260806 10:52:23.817900 4845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:23.817900 4846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:23.819531 4852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:23.825225 4852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:23.833981 4852 catalog_manager.cc:1347] Generated new cluster ID: 8c9ca0a8b6be437a9d0deb79ea44357f
I20260806 10:52:23.834087 4852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:23.857185 4852 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:23.859093 4852 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:23.867676 4852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5da3ab5fddc2401eb19fd9608559bc8c: Generated new TSK 0
I20260806 10:52:23.868767 4852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:23.882931 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4765
W20260806 10:52:23.884670 4829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49979
W20260806 10:52:23.885474 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4765: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:23.885680 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4765
W20260806 10:52:23.903677 4750 connection.cc:535] client connection to 127.0.0.1:33011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33011): Connection reset by peer (error 104)
2026-08-06T10:52:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:24 dist-test-slave-0b58 krb5kdc[4865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:24 dist-test-slave-0b58 krb5kdc[4865](info): set up 2 sockets
Aug 06 10:52:24 dist-test-slave-0b58 krb5kdc[4865](info): commencing operation
krb5kdc: starting...
W20260806 10:52:25.024226 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:25 dist-test-slave-0b58 krb5kdc[4865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:25.332224 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:25.667240 4878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:25.667497 4878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:25.667587 4878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:25.674710 4878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:25.674809 4878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:25.674861 4878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:25.674923 4878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:25.674966 4878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:25.684348 4878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:25.685429 4878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:25.686640 4878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:25.698228 4884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:25.698367 4886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:25.698520 4883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:25.699355 4878 server_base.cc:1034] running on GCE node
I20260806 10:52:25.700045 4878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:25.701411 4878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:25.702611 4878 hybrid_clock.cc:648] HybridClock initialized: now 1786013545702579 us; error 28 us; skew 500 ppm
I20260806 10:52:25.702950 4878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:25 dist-test-slave-0b58 krb5kdc[4865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:25.705951 4878 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:25.707311 4878 webserver.cc:460] Webserver started at http://127.0.0.1:34923/ using document root <none> and password file <none>
I20260806 10:52:25.707954 4878 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:25.708097 4878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:25.708446 4878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:25.711264 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "7bab3fb5e70447f8ab2490360feb6c9d"
format_stamp: "Formatted at 2026-08-06 10:52:25 on dist-test-slave-0b58"
server_key: "588c90b2f55e67e79a06e4c277aa74da"
server_key_iv: "2fff94bf42528f8e0d1e4185453910e2"
server_key_version: "encryptionkey@0"
I20260806 10:52:25.712023 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "7bab3fb5e70447f8ab2490360feb6c9d"
format_stamp: "Formatted at 2026-08-06 10:52:25 on dist-test-slave-0b58"
server_key: "588c90b2f55e67e79a06e4c277aa74da"
server_key_iv: "2fff94bf42528f8e0d1e4185453910e2"
server_key_version: "encryptionkey@0"
I20260806 10:52:25.716929 4878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 10:52:25.720414 4893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:25.721575 4878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:52:25.721753 4878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "7bab3fb5e70447f8ab2490360feb6c9d"
format_stamp: "Formatted at 2026-08-06 10:52:25 on dist-test-slave-0b58"
server_key: "588c90b2f55e67e79a06e4c277aa74da"
server_key_iv: "2fff94bf42528f8e0d1e4185453910e2"
server_key_version: "encryptionkey@0"
I20260806 10:52:25.721941 4878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:25.727313 4878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:25.731595 4878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:25.731914 4878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:25.750667 4878 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42193
I20260806 10:52:25.750770 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42193 every 8 connection(s)
I20260806 10:52:25.752038 4878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:25.755823 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:25.761354 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4878
I20260806 10:52:25.761554 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:25.762101 16468 external_mini_cluster.cc:1428] Setting key 72a6ba98df744dcdb02ccee85d805ef0
I20260806 10:52:25.767846 4946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d: Bootstrap starting.
I20260806 10:52:25.769654 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:25.763889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43875 (local address 127.0.0.1:42193)
0806 10:52:25.764673 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:25.764684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:25.764726 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:52:25.766433 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:25.766453 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:25.766745 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:25.767051 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:25.768150 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:25.768156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:25.768167 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:25.768181 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:25.768244 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:25.768312 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:25.768497 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:25.768600 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 10:52:25.768803 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":380,"threads_started":1}
I20260806 10:52:25.771181 4946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:25.772459 4946 log.cc:824] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:25.773288 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:25.775533 4946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d: No bootstrap required, opened a new log
I20260806 10:52:25.780957 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:25.778398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40449 (local address 127.0.0.1:42193)
0806 10:52:25.778584 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:25.778590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:25.779070 (+ 480us) server_negotiation.cc:407] Connection header received
0806 10:52:25.779189 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:25.779194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:25.779259 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:25.779369 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:25.780240 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:25.780245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:25.780250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:25.780304 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:25.780350 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:25.780374 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:25.780501 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:25.780573 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 10:52:25.780632 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:52:25.784228 4946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260806 10:52:25.784647 4946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:25.784763 4946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bab3fb5e70447f8ab2490360feb6c9d, State: Initialized, Role: FOLLOWER
I20260806 10:52:25.785367 4946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260806 10:52:25.785633 4946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:25.785800 4946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:25.786000 4946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:25.788148 4946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260806 10:52:25.788619 4946 leader_election.cc:302] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bab3fb5e70447f8ab2490360feb6c9d; no voters:
I20260806 10:52:25.789119 4946 leader_election.cc:288] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:25.789244 4956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:25.789984 4956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [term 1 LEADER]: Becoming Leader. State: Replica: 7bab3fb5e70447f8ab2490360feb6c9d, State: Running, Role: LEADER
I20260806 10:52:25.790531 4956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260806 10:52:25.791205 4946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:25.793234 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bab3fb5e70447f8ab2490360feb6c9d. Latest consensus state: current_term: 1 leader_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } } }
I20260806 10:52:25.793627 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:25.794523 4957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bab3fb5e70447f8ab2490360feb6c9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } } }
I20260806 10:52:25.794889 4957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:25.797083 4962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:25.802783 4962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:25.810880 4962 catalog_manager.cc:1347] Generated new cluster ID: c1825a2c2cc0496685356b132bd0a3ee
I20260806 10:52:25.810989 4962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:25.834733 4962 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:25.835145 4962 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:25.849339 4962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bab3fb5e70447f8ab2490360feb6c9d: Generated new TSK 0
I20260806 10:52:25.850363 4962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:25.864416 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4878
W20260806 10:52:25.865875 4941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43875
W20260806 10:52:25.866771 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4878: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:25.866945 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4878
2026-08-06T10:52:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:25 dist-test-slave-0b58 krb5kdc[4978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:25 dist-test-slave-0b58 krb5kdc[4978](info): set up 2 sockets
Aug 06 10:52:25 dist-test-slave-0b58 krb5kdc[4978](info): commencing operation
krb5kdc: starting...
W20260806 10:52:27.004642 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:27 dist-test-slave-0b58 krb5kdc[4978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:27.347702 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:27.681457 4991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:27.681780 4991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:27.681953 4991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:27.689081 4991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:27.689237 4991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:27.689350 4991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:27.689467 4991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:27.689543 4991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:27.699038 4991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:27.700242 4991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:27.701550 4991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:27.712982 4996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:27.713280 4997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:27.713282 4999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:27.714074 4991 server_base.cc:1034] running on GCE node
I20260806 10:52:27.714741 4991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:27.716009 4991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:27.717201 4991 hybrid_clock.cc:648] HybridClock initialized: now 1786013547717134 us; error 65 us; skew 500 ppm
I20260806 10:52:27.717512 4991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:27 dist-test-slave-0b58 krb5kdc[4978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:27.720651 4991 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:27.722041 4991 webserver.cc:460] Webserver started at http://127.0.0.1:33931/ using document root <none> and password file <none>
I20260806 10:52:27.722796 4991 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:27.722932 4991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:27.723241 4991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:27.726243 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "50529c3dc9a34a30967dcec3508845b8"
format_stamp: "Formatted at 2026-08-06 10:52:27 on dist-test-slave-0b58"
server_key: "6c44bace42d6ae1857d47e76e5aa420e"
server_key_iv: "9fe13a2ff46ff04e3fe89e81e1f3a6b1"
server_key_version: "encryptionkey@0"
I20260806 10:52:27.727031 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "50529c3dc9a34a30967dcec3508845b8"
format_stamp: "Formatted at 2026-08-06 10:52:27 on dist-test-slave-0b58"
server_key: "6c44bace42d6ae1857d47e76e5aa420e"
server_key_iv: "9fe13a2ff46ff04e3fe89e81e1f3a6b1"
server_key_version: "encryptionkey@0"
I20260806 10:52:27.731985 4991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:52:27.735446 5006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:27.736811 4991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:52:27.736989 4991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "50529c3dc9a34a30967dcec3508845b8"
format_stamp: "Formatted at 2026-08-06 10:52:27 on dist-test-slave-0b58"
server_key: "6c44bace42d6ae1857d47e76e5aa420e"
server_key_iv: "9fe13a2ff46ff04e3fe89e81e1f3a6b1"
server_key_version: "encryptionkey@0"
I20260806 10:52:27.737193 4991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:27.757679 4991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:27.761643 4991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:27.761930 4991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:27.781594 4991 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45301
I20260806 10:52:27.781626 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45301 every 8 connection(s)
I20260806 10:52:27.782933 4991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:27.786665 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:27.786806 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 4991
I20260806 10:52:27.787012 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:27.787477 16468 external_mini_cluster.cc:1428] Setting key 466e90e468fc84327dfe545ccf806824
I20260806 10:52:27.800567 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8: Bootstrap starting.
Aug 06 10:52:27 dist-test-slave-0b58 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:52:27.804195 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:27.805404 5060 log.cc:824] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:27.809266 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8: No bootstrap required, opened a new log
I20260806 10:52:27.815284 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:27.789288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33979 (local address 127.0.0.1:45301)
0806 10:52:27.790165 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:27.790176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:27.790216 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:52:27.792637 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:27.792662 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:27.793044 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:27.793376 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:27.795172 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:27.796548 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:27.798448 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:27.799107 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:27.799225 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:27.804055 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:27.804062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:27.804075 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:27.804120 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:27.808515 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:27.809508 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:27.809516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:27.809524 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:27.809754 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:27.810290 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:27.810296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:27.810300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:27.811070 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:27.811403 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:27.812808 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0806 10:52:27.813867 (+ 1059us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":403,"threads_started":1}
I20260806 10:52:27.819334 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:27.819447 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50529c3dc9a34a30967dcec3508845b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260806 10:52:27.819854 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:27.819949 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50529c3dc9a34a30967dcec3508845b8, State: Initialized, Role: FOLLOWER
I20260806 10:52:27.820602 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50529c3dc9a34a30967dcec3508845b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260806 10:52:27.820878 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:27.821017 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:27.821170 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:27.824637 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50529c3dc9a34a30967dcec3508845b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260806 10:52:27.825263 5060 leader_election.cc:302] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50529c3dc9a34a30967dcec3508845b8; no voters:
I20260806 10:52:27.825990 5060 leader_election.cc:288] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:27.826079 5070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:27.828523 5070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [term 1 LEADER]: Becoming Leader. State: Replica: 50529c3dc9a34a30967dcec3508845b8, State: Running, Role: LEADER
I20260806 10:52:27.828646 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:27.829164 5070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50529c3dc9a34a30967dcec3508845b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } }
I20260806 10:52:27.832816 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50529c3dc9a34a30967dcec3508845b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50529c3dc9a34a30967dcec3508845b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } } }
I20260806 10:52:27.833710 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:27.836820 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50529c3dc9a34a30967dcec3508845b8. Latest consensus state: current_term: 1 leader_uuid: "50529c3dc9a34a30967dcec3508845b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50529c3dc9a34a30967dcec3508845b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45301 } } }
I20260806 10:52:27.838083 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:27.844044 5080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:27.845686 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:27.824719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53119 (local address 127.0.0.1:45301)
0806 10:52:27.824909 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:27.824915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:27.825585 (+ 670us) server_negotiation.cc:407] Connection header received
0806 10:52:27.825657 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:27.825661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:27.825719 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:27.825839 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:27.827750 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:27.828925 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:27.830585 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:27.831383 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:27.831537 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:27.836042 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:27.836053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:27.836061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:27.836112 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:27.840934 (+ 4822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:27.842100 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:27.842108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:27.842114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:27.842319 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:27.842874 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:27.842882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:27.842888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:27.843500 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:27.843771 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:27.845041 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0806 10:52:27.845306 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:52:27.849886 5080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:27.858664 5080 catalog_manager.cc:1347] Generated new cluster ID: 516ce00a1574403683f28229c3d90954
I20260806 10:52:27.858774 5080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:27.867853 5080 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:27.869099 5080 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:27.878043 5080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50529c3dc9a34a30967dcec3508845b8: Generated new TSK 0
I20260806 10:52:27.878754 5080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:27.887549 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 4991
I20260806 10:52:27.888859 5055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:27.976563 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 4991
2026-08-06T10:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2108 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:28 dist-test-slave-0b58 krb5kdc[5093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:28 dist-test-slave-0b58 krb5kdc[5093](info): set up 2 sockets
Aug 06 10:52:28 dist-test-slave-0b58 krb5kdc[5093](info): commencing operation
krb5kdc: starting...
W20260806 10:52:29.123844 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:29 dist-test-slave-0b58 krb5kdc[5093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:29.447175 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:29.786460 5106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:29.786720 5106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:29.786808 5106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:29.793795 5106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:29.793898 5106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:29.793962 5106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:29.794016 5106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:29.794057 5106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:29.803400 5106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:29.804553 5106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:29.805763 5106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:29.817239 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:29.817256 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:29.817247 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:29.818207 5106 server_base.cc:1034] running on GCE node
I20260806 10:52:29.818964 5106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:29.820335 5106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:29.821559 5106 hybrid_clock.cc:648] HybridClock initialized: now 1786013549821521 us; error 46 us; skew 500 ppm
I20260806 10:52:29.821899 5106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:29 dist-test-slave-0b58 krb5kdc[5093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:29.829056 5106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:29.830703 5106 webserver.cc:460] Webserver started at http://127.0.0.1:35399/ using document root <none> and password file <none>
I20260806 10:52:29.831405 5106 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:29.831557 5106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:29.831881 5106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:29.835026 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "cb9dd3497c014a12823aacf968705c35"
format_stamp: "Formatted at 2026-08-06 10:52:29 on dist-test-slave-0b58"
server_key: "c3cd4b07f16ec0f034c6faa9055ca1bb"
server_key_iv: "d253502664b05b394b53e4ac64f5345a"
server_key_version: "encryptionkey@0"
I20260806 10:52:29.835845 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "cb9dd3497c014a12823aacf968705c35"
format_stamp: "Formatted at 2026-08-06 10:52:29 on dist-test-slave-0b58"
server_key: "c3cd4b07f16ec0f034c6faa9055ca1bb"
server_key_iv: "d253502664b05b394b53e4ac64f5345a"
server_key_version: "encryptionkey@0"
I20260806 10:52:29.841281 5106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 10:52:29.844669 5121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:29.845872 5106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:52:29.846050 5106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "cb9dd3497c014a12823aacf968705c35"
format_stamp: "Formatted at 2026-08-06 10:52:29 on dist-test-slave-0b58"
server_key: "c3cd4b07f16ec0f034c6faa9055ca1bb"
server_key_iv: "d253502664b05b394b53e4ac64f5345a"
server_key_version: "encryptionkey@0"
I20260806 10:52:29.846280 5106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:29.861481 5106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:29.865370 5106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:29.865657 5106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:29.884994 5106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42427
I20260806 10:52:29.885035 5173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42427 every 8 connection(s)
I20260806 10:52:29.886401 5106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:29.889657 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5106
I20260806 10:52:29.889858 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:29.890148 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:29.890337 16468 external_mini_cluster.cc:1428] Setting key e9e7612ddb44eada1eecd0832f768b91
I20260806 10:52:29.905988 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:29.892119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48441 (local address 127.0.0.1:42427)
0806 10:52:29.893106 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:29.893127 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:29.893176 (+ 49us) server_negotiation.cc:407] Connection header received
0806 10:52:29.895324 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:29.895355 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:29.895922 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:29.896398 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:29.896951 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:29.898988 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:29.900341 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:29.901574 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:29.901804 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:29.902932 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:29.902948 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:29.902966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:29.903722 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:29.903830 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:29.903867 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:29.903996 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:29.904226 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 10:52:29.904510 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":378,"threads_started":1}
I20260806 10:52:29.906602 5174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35: Bootstrap starting.
I20260806 10:52:29.909812 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:29.910137 5174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:29.911168 5174 log.cc:824] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:29.914358 5174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35: No bootstrap required, opened a new log
I20260806 10:52:29.921185 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:29.914882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40521 (local address 127.0.0.1:42427)
0806 10:52:29.915072 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:29.915079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:29.915451 (+ 372us) server_negotiation.cc:407] Connection header received
0806 10:52:29.915674 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:29.915678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:29.915745 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:29.915847 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:29.916414 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:29.917183 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:29.918528 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:29.919099 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:29.919276 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:29.920474 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:29.920478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:29.920481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:29.920498 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:29.920534 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:29.920550 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:29.920648 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:29.920796 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 10:52:29.920890 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 10:52:29.922966 5174 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9dd3497c014a12823aacf968705c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260806 10:52:29.923337 5174 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:29.923446 5174 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9dd3497c014a12823aacf968705c35, State: Initialized, Role: FOLLOWER
I20260806 10:52:29.923976 5174 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9dd3497c014a12823aacf968705c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260806 10:52:29.924226 5174 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:29.924436 5174 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:29.924611 5174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:29.926944 5174 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9dd3497c014a12823aacf968705c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260806 10:52:29.927532 5174 leader_election.cc:302] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb9dd3497c014a12823aacf968705c35; no voters:
I20260806 10:52:29.928175 5174 leader_election.cc:288] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:29.928330 5184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:29.929098 5184 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [term 1 LEADER]: Becoming Leader. State: Replica: cb9dd3497c014a12823aacf968705c35, State: Running, Role: LEADER
I20260806 10:52:29.929733 5184 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9dd3497c014a12823aacf968705c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260806 10:52:29.930420 5174 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:29.932639 5186 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb9dd3497c014a12823aacf968705c35. Latest consensus state: current_term: 1 leader_uuid: "cb9dd3497c014a12823aacf968705c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9dd3497c014a12823aacf968705c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } } }
I20260806 10:52:29.933091 5186 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:29.933676 5185 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb9dd3497c014a12823aacf968705c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9dd3497c014a12823aacf968705c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } } }
I20260806 10:52:29.934019 5185 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:29.937620 5192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:29.942636 5192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:29.951792 5192 catalog_manager.cc:1347] Generated new cluster ID: 86f83ef0a3e84d5baeb6eaf6cce0e1b7
I20260806 10:52:29.951921 5192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:29.971935 5192 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:29.973240 5192 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:29.983517 5192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb9dd3497c014a12823aacf968705c35: Generated new TSK 0
I20260806 10:52:29.984324 5192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:29.992488 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5106
W20260806 10:52:29.993455 5169 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48441
W20260806 10:52:29.994098 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5106: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:29.994256 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5106
2026-08-06T10:52:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:30 dist-test-slave-0b58 krb5kdc[5205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:30 dist-test-slave-0b58 krb5kdc[5205](info): set up 2 sockets
Aug 06 10:52:30 dist-test-slave-0b58 krb5kdc[5205](info): commencing operation
krb5kdc: starting...
W20260806 10:52:31.140375 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:31 dist-test-slave-0b58 krb5kdc[5205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:31.460047 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:31.795503 5218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:31.795771 5218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:31.795893 5218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:31.802930 5218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:31.803051 5218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:31.803167 5218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:31.803264 5218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:31.803352 5218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:31.812698 5218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:31.813843 5218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:31.815047 5218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:31.826690 5226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:31.826637 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:31.826742 5223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:31.827512 5218 server_base.cc:1034] running on GCE node
I20260806 10:52:31.828217 5218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:31.829546 5218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:31.830746 5218 hybrid_clock.cc:648] HybridClock initialized: now 1786013551830714 us; error 24 us; skew 500 ppm
I20260806 10:52:31.831049 5218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:31 dist-test-slave-0b58 krb5kdc[5205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:31.834185 5218 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:31.835525 5218 webserver.cc:460] Webserver started at http://127.0.0.1:43367/ using document root <none> and password file <none>
I20260806 10:52:31.836088 5218 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:31.836197 5218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:31.836505 5218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:31.839385 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "e45efbc6b6f840f890a4a64bd4fbd151"
format_stamp: "Formatted at 2026-08-06 10:52:31 on dist-test-slave-0b58"
server_key: "3d933b530206b528551af236d6759bfe"
server_key_iv: "99152e92a147b522dad76b6b47ee863e"
server_key_version: "encryptionkey@0"
I20260806 10:52:31.840094 5218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "e45efbc6b6f840f890a4a64bd4fbd151"
format_stamp: "Formatted at 2026-08-06 10:52:31 on dist-test-slave-0b58"
server_key: "3d933b530206b528551af236d6759bfe"
server_key_iv: "99152e92a147b522dad76b6b47ee863e"
server_key_version: "encryptionkey@0"
I20260806 10:52:31.844931 5218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 10:52:31.848091 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:31.849417 5218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:52:31.849551 5218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "e45efbc6b6f840f890a4a64bd4fbd151"
format_stamp: "Formatted at 2026-08-06 10:52:31 on dist-test-slave-0b58"
server_key: "3d933b530206b528551af236d6759bfe"
server_key_iv: "99152e92a147b522dad76b6b47ee863e"
server_key_version: "encryptionkey@0"
I20260806 10:52:31.849718 5218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:31.855208 5218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:31.859464 5218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:31.859714 5218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:31.878825 5218 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45599
I20260806 10:52:31.878870 5285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45599 every 8 connection(s)
I20260806 10:52:31.880235 5218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:31.883885 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:31.889781 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5218
I20260806 10:52:31.889976 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:31.890558 16468 external_mini_cluster.cc:1428] Setting key 17b91179282c9f027f30d81cfc5fb1d4
I20260806 10:52:31.894897 5286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151: Bootstrap starting.
I20260806 10:52:31.898556 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:31.892594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45631 (local address 127.0.0.1:45599)
0806 10:52:31.893494 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:31.893505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:31.893545 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:52:31.895268 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:31.895290 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:31.895610 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:31.895951 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:31.897035 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:31.897041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:31.897053 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:31.897067 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:31.897132 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:31.897150 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:31.897319 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:31.897464 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 10:52:31.897679 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":404,"threads_started":1}
I20260806 10:52:31.898645 5286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:31.899796 5286 log.cc:824] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:31.902362 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:31.902839 5286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151: No bootstrap required, opened a new log
I20260806 10:52:31.909368 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:31.906838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54133 (local address 127.0.0.1:45599)
0806 10:52:31.907028 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:31.907036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:31.907345 (+ 309us) server_negotiation.cc:407] Connection header received
0806 10:52:31.907548 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:31.907552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:31.907600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:31.907711 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:31.908683 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:31.908687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:31.908690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:31.908705 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:31.908743 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:31.908766 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:31.908898 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:31.909017 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 10:52:31.909076 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:52:31.911866 5286 raft_consensus.cc:348] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260806 10:52:31.912299 5286 raft_consensus.cc:374] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:31.912418 5286 raft_consensus.cc:729] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45efbc6b6f840f890a4a64bd4fbd151, State: Initialized, Role: FOLLOWER
I20260806 10:52:31.913111 5286 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260806 10:52:31.913381 5286 raft_consensus.cc:388] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:31.913545 5286 raft_consensus.cc:482] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:31.913723 5286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:31.916100 5286 raft_consensus.cc:504] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260806 10:52:31.916703 5286 leader_election.cc:302] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e45efbc6b6f840f890a4a64bd4fbd151; no voters:
I20260806 10:52:31.917460 5286 leader_election.cc:288] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:31.917603 5296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:31.918300 5296 raft_consensus.cc:686] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [term 1 LEADER]: Becoming Leader. State: Replica: e45efbc6b6f840f890a4a64bd4fbd151, State: Running, Role: LEADER
I20260806 10:52:31.918813 5296 consensus_queue.cc:237] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260806 10:52:31.919560 5286 sys_catalog.cc:564] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:31.921880 5297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } } }
I20260806 10:52:31.921854 5298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45efbc6b6f840f890a4a64bd4fbd151. Latest consensus state: current_term: 1 leader_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45efbc6b6f840f890a4a64bd4fbd151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } } }
I20260806 10:52:31.922340 5297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:31.922358 5298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:31.923991 5301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:31.929497 5301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:31.938239 5301 catalog_manager.cc:1347] Generated new cluster ID: b305d4fdac2a4fbfaa272c39a3d7073e
I20260806 10:52:31.938352 5301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:31.960357 5301 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:31.960740 5301 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:31.972653 5301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e45efbc6b6f840f890a4a64bd4fbd151: Generated new TSK 0
I20260806 10:52:31.973563 5301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:31.990094 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5218
W20260806 10:52:31.991837 5281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45631
W20260806 10:52:31.992539 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:31.992686 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5218
2026-08-06T10:52:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:32 dist-test-slave-0b58 krb5kdc[5318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:32 dist-test-slave-0b58 krb5kdc[5318](info): set up 2 sockets
Aug 06 10:52:32 dist-test-slave-0b58 krb5kdc[5318](info): commencing operation
krb5kdc: starting...
W20260806 10:52:33.124194 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.004s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:33 dist-test-slave-0b58 krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:33.439878 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:33.770505 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:33.770831 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:33.771003 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:33.777935 5331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:33.778079 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:33.778179 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:33.778307 5331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:33.778380 5331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:33.787798 5331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:33.789070 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:33.790279 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:33.801527 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:33.801563 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:33.802230 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:33.802699 5331 server_base.cc:1034] running on GCE node
I20260806 10:52:33.803380 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:33.804612 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:33.805819 5331 hybrid_clock.cc:648] HybridClock initialized: now 1786013553805771 us; error 24 us; skew 500 ppm
I20260806 10:52:33.806145 5331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:33 dist-test-slave-0b58 krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:33.809250 5331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:33.810621 5331 webserver.cc:460] Webserver started at http://127.0.0.1:41999/ using document root <none> and password file <none>
I20260806 10:52:33.811285 5331 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:33.811421 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:33.811714 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:33.814599 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "82e0b79b715c43bd856047b742fdeb5b"
format_stamp: "Formatted at 2026-08-06 10:52:33 on dist-test-slave-0b58"
server_key: "0350781ead894a3df6c9816558caae33"
server_key_iv: "5fc7aff5a6e9e720eeb42e94859f1c34"
server_key_version: "encryptionkey@0"
I20260806 10:52:33.815372 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "82e0b79b715c43bd856047b742fdeb5b"
format_stamp: "Formatted at 2026-08-06 10:52:33 on dist-test-slave-0b58"
server_key: "0350781ead894a3df6c9816558caae33"
server_key_iv: "5fc7aff5a6e9e720eeb42e94859f1c34"
server_key_version: "encryptionkey@0"
I20260806 10:52:33.820086 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260806 10:52:33.823344 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:33.824669 5331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:33.824863 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "82e0b79b715c43bd856047b742fdeb5b"
format_stamp: "Formatted at 2026-08-06 10:52:33 on dist-test-slave-0b58"
server_key: "0350781ead894a3df6c9816558caae33"
server_key_iv: "5fc7aff5a6e9e720eeb42e94859f1c34"
server_key_version: "encryptionkey@0"
I20260806 10:52:33.825050 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:33.839541 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:33.843333 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:33.843621 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:33.862478 5331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41143
I20260806 10:52:33.862519 5397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41143 every 8 connection(s)
I20260806 10:52:33.863752 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:33.867154 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:33.869764 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5331
I20260806 10:52:33.869946 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:33.870463 16468 external_mini_cluster.cc:1428] Setting key 297a523487a36017dce3ab4f72e08419
I20260806 10:52:33.879191 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b: Bootstrap starting.
I20260806 10:52:33.882995 5398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:33.884388 5398 log.cc:824] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:52:33 dist-test-slave-0b58 krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:52:33.887357 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b: No bootstrap required, opened a new log
I20260806 10:52:33.893803 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:33.872303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:47427 (local address 127.0.0.1:41143)
0806 10:52:33.873118 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:33.873128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:33.873167 (+ 39us) server_negotiation.cc:407] Connection header received
0806 10:52:33.874783 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:33.874802 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:33.875145 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:33.875449 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:33.877197 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:33.878507 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:33.880309 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:33.880979 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:33.881101 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:33.886495 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:33.886503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:33.886516 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:33.886550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:33.889695 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:33.890531 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:33.890537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:33.890542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:33.890704 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:33.891185 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:33.891188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:33.891190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:33.891712 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:33.892062 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:33.892565 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 10:52:33.892874 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":390,"threads_started":1}
I20260806 10:52:33.895490 5398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e0b79b715c43bd856047b742fdeb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260806 10:52:33.895852 5398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:33.895977 5398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e0b79b715c43bd856047b742fdeb5b, State: Initialized, Role: FOLLOWER
I20260806 10:52:33.896657 5398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e0b79b715c43bd856047b742fdeb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260806 10:52:33.896935 5398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:33.897082 5398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:33.897269 5398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:33.897542 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:33.900388 5398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e0b79b715c43bd856047b742fdeb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260806 10:52:33.900974 5398 leader_election.cc:302] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e0b79b715c43bd856047b742fdeb5b; no voters:
I20260806 10:52:33.901860 5398 leader_election.cc:288] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:33.901968 5406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:33.904314 5398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:33.904563 5406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [term 1 LEADER]: Becoming Leader. State: Replica: 82e0b79b715c43bd856047b742fdeb5b, State: Running, Role: LEADER
I20260806 10:52:33.905182 5406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e0b79b715c43bd856047b742fdeb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260806 10:52:33.908743 5409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e0b79b715c43bd856047b742fdeb5b. Latest consensus state: current_term: 1 leader_uuid: "82e0b79b715c43bd856047b742fdeb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e0b79b715c43bd856047b742fdeb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260806 10:52:33.909155 5409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:33.909642 5408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e0b79b715c43bd856047b742fdeb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e0b79b715c43bd856047b742fdeb5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260806 10:52:33.910054 5408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:33.912155 5417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:33.917977 5417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:33.924610 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:33.903307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:54827 (local address 127.0.0.1:41143)
0806 10:52:33.903535 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:33.903554 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:33.904586 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 10:52:33.904998 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:33.905004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:33.905095 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:33.905208 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:33.907843 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:33.909452 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:33.912787 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:33.913478 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:33.913577 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:33.917082 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:33.917089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:33.917097 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:33.917136 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:33.920881 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:33.921881 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:33.921887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:33.921891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:33.922045 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:33.922623 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:33.922629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:33.922645 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:33.923086 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:33.923359 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:33.924041 (+ 682us) server_negotiation.cc:299] Negotiation successful
0806 10:52:33.924216 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260806 10:52:33.929013 5417 catalog_manager.cc:1347] Generated new cluster ID: d583db5f556d411db62cb800c8b7cb1a
I20260806 10:52:33.929112 5417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:33.937623 5417 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:33.938834 5417 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:33.957888 5417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e0b79b715c43bd856047b742fdeb5b: Generated new TSK 0
I20260806 10:52:33.958618 5417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:33.969441 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5331
I20260806 10:52:33.971563 5393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:34.061993 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5331
W20260806 10:52:34.081956 5316 connection.cc:535] client connection to 127.0.0.1:41143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41143): Connection reset by peer (error 104)
2026-08-06T10:52:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:34 dist-test-slave-0b58 krb5kdc[5431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:34 dist-test-slave-0b58 krb5kdc[5431](info): set up 2 sockets
Aug 06 10:52:34 dist-test-slave-0b58 krb5kdc[5431](info): commencing operation
krb5kdc: starting...
W20260806 10:52:35.188403 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:35 dist-test-slave-0b58 krb5kdc[5431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:35.493732 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:35.826341 5444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:35.826597 5444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:35.826758 5444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:35.833647 5444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:35.833773 5444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:35.833866 5444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:35.833958 5444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:35.834035 5444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:35.843299 5444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:35.844491 5444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:35.845726 5444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:35.857366 5449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:35.857358 5452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:35.857380 5450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:35.858181 5444 server_base.cc:1034] running on GCE node
I20260806 10:52:35.859009 5444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:35.860363 5444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:35.861589 5444 hybrid_clock.cc:648] HybridClock initialized: now 1786013555861545 us; error 51 us; skew 500 ppm
I20260806 10:52:35.861941 5444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:35 dist-test-slave-0b58 krb5kdc[5431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:35.868834 5444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:35.870143 5444 webserver.cc:460] Webserver started at http://127.0.0.1:44327/ using document root <none> and password file <none>
I20260806 10:52:35.870733 5444 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:35.870839 5444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:35.871073 5444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:35.873975 5444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc"
format_stamp: "Formatted at 2026-08-06 10:52:35 on dist-test-slave-0b58"
server_key: "a08a1dcc855c4b459525cb28e8c45c00"
server_key_iv: "b148a5a8143048475ce95c8e8942ac4d"
server_key_version: "encryptionkey@0"
I20260806 10:52:35.874709 5444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc"
format_stamp: "Formatted at 2026-08-06 10:52:35 on dist-test-slave-0b58"
server_key: "a08a1dcc855c4b459525cb28e8c45c00"
server_key_iv: "b148a5a8143048475ce95c8e8942ac4d"
server_key_version: "encryptionkey@0"
I20260806 10:52:35.879541 5444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:52:35.882809 5459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:35.884112 5444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:52:35.884245 5444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc"
format_stamp: "Formatted at 2026-08-06 10:52:35 on dist-test-slave-0b58"
server_key: "a08a1dcc855c4b459525cb28e8c45c00"
server_key_iv: "b148a5a8143048475ce95c8e8942ac4d"
server_key_version: "encryptionkey@0"
I20260806 10:52:35.884462 5444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:35.899973 5444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:35.903841 5444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:35.904074 5444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:35.922583 5444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44361
I20260806 10:52:35.922649 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44361 every 8 connection(s)
I20260806 10:52:35.923867 5444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:35.924278 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5444
I20260806 10:52:35.924492 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:35.925086 16468 external_mini_cluster.cc:1428] Setting key 8aa037e6af76616fbf0fe102c2ee762a
I20260806 10:52:35.928203 5512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:35.938330 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:35.926916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:47913 (local address 127.0.0.1:44361)
0806 10:52:35.927732 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:35.927743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:35.927781 (+ 38us) server_negotiation.cc:407] Connection header received
0806 10:52:35.929505 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:35.929525 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:35.929897 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:35.930185 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:35.930750 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:35.932083 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:35.934066 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:35.934721 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:35.934850 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:35.936076 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:35.936084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:35.936095 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:35.936655 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:35.936727 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:35.936750 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:35.936874 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:35.937211 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 10:52:35.937387 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":373,"threads_started":1}
I20260806 10:52:35.939957 5512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc: Bootstrap starting.
I20260806 10:52:35.941788 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:35.943255 5512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:35.944535 5512 log.cc:824] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:35.947221 5512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc: No bootstrap required, opened a new log
I20260806 10:52:35.953722 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:35.947227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:55633 (local address 127.0.0.1:44361)
0806 10:52:35.947402 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:35.947407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:35.947803 (+ 396us) server_negotiation.cc:407] Connection header received
0806 10:52:35.947940 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:35.947947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:35.948015 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:35.948143 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:35.948694 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:35.949855 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:35.951128 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:35.951817 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:35.951956 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:35.952980 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:35.952986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:35.952991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:35.953009 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:35.953053 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:35.953077 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:35.953194 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:35.953342 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 10:52:35.953389 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:52:35.956676 5512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44361 } }
I20260806 10:52:35.956979 5512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:35.957094 5512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2fe480e9c54f9d9b8b4d5987fd5bfc, State: Initialized, Role: FOLLOWER
I20260806 10:52:35.957696 5512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44361 } }
I20260806 10:52:35.957991 5512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:35.958159 5512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:35.958369 5512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:35.960526 5512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44361 } }
I20260806 10:52:35.961076 5512 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2fe480e9c54f9d9b8b4d5987fd5bfc; no voters:
I20260806 10:52:35.961710 5512 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:35.961856 5522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:35.962566 5522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [term 1 LEADER]: Becoming Leader. State: Replica: 5b2fe480e9c54f9d9b8b4d5987fd5bfc, State: Running, Role: LEADER
I20260806 10:52:35.963101 5522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44361 } }
I20260806 10:52:35.963673 5512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:35.966218 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44361 } } }
I20260806 10:52:35.966364 5524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2fe480e9c54f9d9b8b4d5987fd5bfc. Latest consensus state: current_term: 1 leader_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fe480e9c54f9d9b8b4d5987fd5bfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44361 } } }
I20260806 10:52:35.966712 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:35.966706 5524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:35.968889 5531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:35.973388 5531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:35.981540 5531 catalog_manager.cc:1347] Generated new cluster ID: d7c079f57c00454aa4ce672b45d10536
I20260806 10:52:35.981662 5531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:35.996558 5531 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:35.997704 5531 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:36.006852 5531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b2fe480e9c54f9d9b8b4d5987fd5bfc: Generated new TSK 0
I20260806 10:52:36.007588 5531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:36.021006 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5444
W20260806 10:52:36.022228 5507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.101:47913
W20260806 10:52:36.023001 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5444: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:36.023159 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5444
W20260806 10:52:36.044806 5429 connection.cc:535] client connection to 127.0.0.1:44361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44361): Connection reset by peer (error 104)
2026-08-06T10:52:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:36 dist-test-slave-0b58 krb5kdc[5543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:36 dist-test-slave-0b58 krb5kdc[5543](info): set up 2 sockets
Aug 06 10:52:36 dist-test-slave-0b58 krb5kdc[5543](info): commencing operation
krb5kdc: starting...
W20260806 10:52:37.148841 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.005s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:37 dist-test-slave-0b58 krb5kdc[5543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:37.451032 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:37.768560 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:37.768796 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:37.768956 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:37.775732 5556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:37.775820 5556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:37.775861 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:37.775949 5556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:37.776012 5556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:37.785109 5556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:37.786199 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:37.787454 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:37.798501 5562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:37.798591 5561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:37.798641 5564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:37.799081 5556 server_base.cc:1034] running on GCE node
I20260806 10:52:37.799716 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:37.801024 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:37.802222 5556 hybrid_clock.cc:648] HybridClock initialized: now 1786013557802175 us; error 38 us; skew 500 ppm
I20260806 10:52:37.802572 5556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:37 dist-test-slave-0b58 krb5kdc[5543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:37.805622 5556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:37.806962 5556 webserver.cc:460] Webserver started at http://127.0.0.1:36759/ using document root <none> and password file <none>
I20260806 10:52:37.807621 5556 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:37.807749 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:37.808091 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:37.810977 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "7ee6774e1b6942c588188b47b2bf53ce"
format_stamp: "Formatted at 2026-08-06 10:52:37 on dist-test-slave-0b58"
server_key: "3d4785a5006d823c4695becb9b0a0e25"
server_key_iv: "9a69eb075549a26a15d4409aacc7eb7b"
server_key_version: "encryptionkey@0"
I20260806 10:52:37.811733 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "7ee6774e1b6942c588188b47b2bf53ce"
format_stamp: "Formatted at 2026-08-06 10:52:37 on dist-test-slave-0b58"
server_key: "3d4785a5006d823c4695becb9b0a0e25"
server_key_iv: "9a69eb075549a26a15d4409aacc7eb7b"
server_key_version: "encryptionkey@0"
I20260806 10:52:37.816614 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 10:52:37.819644 5571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:37.820950 5556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:37.821136 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "7ee6774e1b6942c588188b47b2bf53ce"
format_stamp: "Formatted at 2026-08-06 10:52:37 on dist-test-slave-0b58"
server_key: "3d4785a5006d823c4695becb9b0a0e25"
server_key_iv: "9a69eb075549a26a15d4409aacc7eb7b"
server_key_version: "encryptionkey@0"
I20260806 10:52:37.821352 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:37.826825 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:37.833104 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:37.833468 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:37.851648 5556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46585
I20260806 10:52:37.851722 5622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46585 every 8 connection(s)
I20260806 10:52:37.853026 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:37.855618 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5556
I20260806 10:52:37.855808 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:37.856314 16468 external_mini_cluster.cc:1428] Setting key 176daf8f2a47a8166cbf94e1b120240f
I20260806 10:52:37.856472 5623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:37.864580 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:37.858120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:60413 (local address 127.0.0.1:46585)
0806 10:52:37.858929 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:37.858941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:37.858983 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:52:37.861032 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:37.861056 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:37.861446 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:37.861869 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:37.862811 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:37.862820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:37.862836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:37.862879 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:37.862957 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:37.862986 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:37.863181 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:37.863311 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 10:52:37.863485 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":300,"threads_started":1}
I20260806 10:52:37.868147 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:37.870715 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce: Bootstrap starting.
I20260806 10:52:37.874493 5623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:37.875604 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:37.873164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:53661 (local address 127.0.0.1:46585)
0806 10:52:37.873335 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:37.873341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:37.873839 (+ 498us) server_negotiation.cc:407] Connection header received
0806 10:52:37.873972 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:37.873976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:37.874013 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:37.874082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:37.875000 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:37.875004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:37.875007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:37.875020 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:37.875052 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:37.875064 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:37.875145 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:37.875284 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 10:52:37.875324 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:52:37.875753 5623 log.cc:824] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:37.878355 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce: No bootstrap required, opened a new log
I20260806 10:52:37.886054 5623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee6774e1b6942c588188b47b2bf53ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260806 10:52:37.886438 5623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:37.886555 5623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee6774e1b6942c588188b47b2bf53ce, State: Initialized, Role: FOLLOWER
I20260806 10:52:37.887063 5623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee6774e1b6942c588188b47b2bf53ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260806 10:52:37.887296 5623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:37.887470 5623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:37.887645 5623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:37.889789 5623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee6774e1b6942c588188b47b2bf53ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260806 10:52:37.890228 5623 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ee6774e1b6942c588188b47b2bf53ce; no voters:
I20260806 10:52:37.890926 5623 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:37.891057 5633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:37.891813 5633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [term 1 LEADER]: Becoming Leader. State: Replica: 7ee6774e1b6942c588188b47b2bf53ce, State: Running, Role: LEADER
I20260806 10:52:37.892407 5633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee6774e1b6942c588188b47b2bf53ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } }
I20260806 10:52:37.893281 5623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:37.895010 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ee6774e1b6942c588188b47b2bf53ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee6774e1b6942c588188b47b2bf53ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } } }
I20260806 10:52:37.895444 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:37.897112 5635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee6774e1b6942c588188b47b2bf53ce. Latest consensus state: current_term: 1 leader_uuid: "7ee6774e1b6942c588188b47b2bf53ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee6774e1b6942c588188b47b2bf53ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46585 } } }
I20260806 10:52:37.897337 5639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:37.897354 5635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:37.908736 5639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:37.916812 5639 catalog_manager.cc:1347] Generated new cluster ID: e3d1d15ee48143299c06e465bc25898a
I20260806 10:52:37.916937 5639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:37.932647 5639 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:37.933097 5639 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:37.940737 5639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ee6774e1b6942c588188b47b2bf53ce: Generated new TSK 0
I20260806 10:52:37.941502 5639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:37.950294 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5556
W20260806 10:52:37.951342 5618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.101:60413
W20260806 10:52:37.952000 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:37.952139 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5556
2026-08-06T10:52:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:38 dist-test-slave-0b58 krb5kdc[5655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:38 dist-test-slave-0b58 krb5kdc[5655](info): set up 2 sockets
Aug 06 10:52:38 dist-test-slave-0b58 krb5kdc[5655](info): commencing operation
krb5kdc: starting...
W20260806 10:52:39.052038 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:39 dist-test-slave-0b58 krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:39.356433 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:39.692001 5668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:39.692417 5668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:39.692588 5668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:39.699455 5668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:39.699605 5668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:39.699713 5668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:39.699823 5668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:39.699905 5668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:39.709549 5668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:39.710724 5668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:39.711946 5668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:39.723538 5674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:39.723578 5675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:39.723501 5677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:39.724785 5668 server_base.cc:1034] running on GCE node
I20260806 10:52:39.725503 5668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:39.726817 5668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:39.728047 5668 hybrid_clock.cc:648] HybridClock initialized: now 1786013559728013 us; error 42 us; skew 500 ppm
I20260806 10:52:39.728466 5668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:39 dist-test-slave-0b58 krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:39.731503 5668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:39.732983 5668 webserver.cc:460] Webserver started at http://127.0.0.1:41153/ using document root <none> and password file <none>
I20260806 10:52:39.733596 5668 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:39.733700 5668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:39.733933 5668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:39.737237 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "a68d4027e50e4064add631cad1c252ce"
format_stamp: "Formatted at 2026-08-06 10:52:39 on dist-test-slave-0b58"
server_key: "567acfdebf130f9b20295bf12a25f144"
server_key_iv: "76c3bae6e1b2bff2c54a40981735dd43"
server_key_version: "encryptionkey@0"
I20260806 10:52:39.738082 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "a68d4027e50e4064add631cad1c252ce"
format_stamp: "Formatted at 2026-08-06 10:52:39 on dist-test-slave-0b58"
server_key: "567acfdebf130f9b20295bf12a25f144"
server_key_iv: "76c3bae6e1b2bff2c54a40981735dd43"
server_key_version: "encryptionkey@0"
I20260806 10:52:39.742832 5668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:52:39.746013 5684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:39.747257 5668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 10:52:39.747406 5668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "a68d4027e50e4064add631cad1c252ce"
format_stamp: "Formatted at 2026-08-06 10:52:39 on dist-test-slave-0b58"
server_key: "567acfdebf130f9b20295bf12a25f144"
server_key_iv: "76c3bae6e1b2bff2c54a40981735dd43"
server_key_version: "encryptionkey@0"
I20260806 10:52:39.747556 5668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:39.768952 5668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:39.773079 5668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:39.773437 5668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:39.791795 5668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39061
I20260806 10:52:39.791842 5735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39061 every 8 connection(s)
I20260806 10:52:39.793210 5668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:39.796944 5736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:39.798681 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5668
I20260806 10:52:39.798890 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:39.799345 16468 external_mini_cluster.cc:1428] Setting key 7c50e5f4953925b10a0371db000fdb6e
I20260806 10:52:39.809387 5736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce: Bootstrap starting.
Aug 06 10:52:39 dist-test-slave-0b58 krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:52:39.812584 5736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:39.813585 5736 log.cc:824] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:39.816630 5736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce: No bootstrap required, opened a new log
I20260806 10:52:39.821135 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:39.801135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:33595 (local address 127.0.0.1:39061)
0806 10:52:39.801830 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:39.801844 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:39.801882 (+ 38us) server_negotiation.cc:407] Connection header received
0806 10:52:39.803464 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:39.803487 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:39.803858 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:39.804228 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:39.805683 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:39.806997 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:39.808250 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:39.808943 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:39.809164 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:39.813417 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:39.813424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:39.813439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:39.813471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:39.816792 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:39.817710 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:39.817716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:39.817721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:39.817897 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:39.818442 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:39.818446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:39.818448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:39.818947 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:39.819191 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:39.819696 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 10:52:39.820025 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":285,"threads_started":1}
I20260806 10:52:39.824975 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:39.825780 5736 raft_consensus.cc:348] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d4027e50e4064add631cad1c252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260806 10:52:39.826225 5736 raft_consensus.cc:374] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:39.826333 5736 raft_consensus.cc:729] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68d4027e50e4064add631cad1c252ce, State: Initialized, Role: FOLLOWER
I20260806 10:52:39.826951 5736 consensus_queue.cc:260] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d4027e50e4064add631cad1c252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260806 10:52:39.827224 5736 raft_consensus.cc:388] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:39.827360 5736 raft_consensus.cc:482] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:39.827566 5736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:39.830901 5736 raft_consensus.cc:504] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d4027e50e4064add631cad1c252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260806 10:52:39.831485 5736 leader_election.cc:302] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a68d4027e50e4064add631cad1c252ce; no voters:
I20260806 10:52:39.832192 5736 leader_election.cc:288] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:39.832315 5746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:39.834394 5736 sys_catalog.cc:564] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:39.834513 5746 raft_consensus.cc:686] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [term 1 LEADER]: Becoming Leader. State: Replica: a68d4027e50e4064add631cad1c252ce, State: Running, Role: LEADER
I20260806 10:52:39.835144 5746 consensus_queue.cc:237] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d4027e50e4064add631cad1c252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260806 10:52:39.839042 5748 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a68d4027e50e4064add631cad1c252ce. Latest consensus state: current_term: 1 leader_uuid: "a68d4027e50e4064add631cad1c252ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d4027e50e4064add631cad1c252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } } }
I20260806 10:52:39.839449 5748 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:39.840015 5747 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a68d4027e50e4064add631cad1c252ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d4027e50e4064add631cad1c252ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } } }
I20260806 10:52:39.840350 5747 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:39.842876 5755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:39.848750 5755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:39.850607 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:39.830294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:42431 (local address 127.0.0.1:39061)
0806 10:52:39.830534 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:39.830543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:39.830956 (+ 413us) server_negotiation.cc:407] Connection header received
0806 10:52:39.831137 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:39.831145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:39.831218 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:39.831295 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:39.833416 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:39.834906 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:39.837036 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:39.837964 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:39.838095 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:39.841967 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:39.841977 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:39.841985 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:39.842043 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:39.846411 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:39.847426 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:39.847432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:39.847438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:39.847601 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:39.848162 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:39.848167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:39.848170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:39.848654 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:39.848963 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:39.850085 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0806 10:52:39.850260 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:52:39.857769 5755 catalog_manager.cc:1347] Generated new cluster ID: e93a013a2b7047799c5d282111015c23
I20260806 10:52:39.857882 5755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:39.881968 5755 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:39.883713 5755 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:39.895552 5755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a68d4027e50e4064add631cad1c252ce: Generated new TSK 0
I20260806 10:52:39.896570 5755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:39.903591 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5668
I20260806 10:52:39.904870 5731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:39.992846 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5668
W20260806 10:52:40.012821 5653 connection.cc:535] client connection to 127.0.0.1:39061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39061): Connection reset by peer (error 104)
2026-08-06T10:52:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:40 dist-test-slave-0b58 krb5kdc[5768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:40 dist-test-slave-0b58 krb5kdc[5768](info): set up 2 sockets
Aug 06 10:52:40 dist-test-slave-0b58 krb5kdc[5768](info): commencing operation
krb5kdc: starting...
W20260806 10:52:41.120018 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:41 dist-test-slave-0b58 krb5kdc[5768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:41.463269 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:41.793787 5781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:41.794103 5781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:41.794267 5781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:41.801013 5781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:41.801144 5781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:41.801256 5781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:41.801359 5781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:41.801443 5781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:41.810756 5781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:41.811892 5781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:41.813141 5781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:41.824688 5787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:41.824728 5789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:41.824749 5786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:41.825546 5781 server_base.cc:1034] running on GCE node
I20260806 10:52:41.826247 5781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:41.827474 5781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:41.828709 5781 hybrid_clock.cc:648] HybridClock initialized: now 1786013561828668 us; error 50 us; skew 500 ppm
I20260806 10:52:41.829056 5781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:41 dist-test-slave-0b58 krb5kdc[5768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:41.836597 5781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:41.837960 5781 webserver.cc:460] Webserver started at http://127.0.0.1:44511/ using document root <none> and password file <none>
I20260806 10:52:41.838649 5781 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:41.838812 5781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:41.839107 5781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:41.841990 5781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "a2a3cd09e67a46a284d228d940060830"
format_stamp: "Formatted at 2026-08-06 10:52:41 on dist-test-slave-0b58"
server_key: "12b4ca771241625b07160b13b2114151"
server_key_iv: "01b0a5db7630d09f0b7f1e7dd0245219"
server_key_version: "encryptionkey@0"
I20260806 10:52:41.842837 5781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "a2a3cd09e67a46a284d228d940060830"
format_stamp: "Formatted at 2026-08-06 10:52:41 on dist-test-slave-0b58"
server_key: "12b4ca771241625b07160b13b2114151"
server_key_iv: "01b0a5db7630d09f0b7f1e7dd0245219"
server_key_version: "encryptionkey@0"
I20260806 10:52:41.847810 5781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 10:52:41.851161 5796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:41.852460 5781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:41.852633 5781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "a2a3cd09e67a46a284d228d940060830"
format_stamp: "Formatted at 2026-08-06 10:52:41 on dist-test-slave-0b58"
server_key: "12b4ca771241625b07160b13b2114151"
server_key_iv: "01b0a5db7630d09f0b7f1e7dd0245219"
server_key_version: "encryptionkey@0"
I20260806 10:52:41.852831 5781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:41.864921 5781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:41.869093 5781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:41.869381 5781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:41.888880 5781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46083
I20260806 10:52:41.888918 5849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46083 every 8 connection(s)
I20260806 10:52:41.890264 5781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:41.893270 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5781
I20260806 10:52:41.893497 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:41.893967 16468 external_mini_cluster.cc:1428] Setting key 389ee05d386b48712d3c2139983b6b7b
I20260806 10:52:41.893977 5850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:41.908620 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:41.895674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:56535 (local address 127.0.0.1:46083)
0806 10:52:41.896442 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:41.896457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:41.896512 (+ 55us) server_negotiation.cc:407] Connection header received
0806 10:52:41.898766 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:41.898793 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:41.899302 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:41.899704 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:41.900221 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:41.902204 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:41.903559 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:41.904592 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:41.904778 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:41.905893 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:41.905908 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:41.905925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:41.906671 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:41.906769 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:41.906798 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:41.906945 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:41.907184 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 10:52:41.907333 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":295,"threads_started":1}
I20260806 10:52:41.908921 5850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830: Bootstrap starting.
I20260806 10:52:41.911916 5850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:41.912216 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:41.913198 5850 log.cc:824] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:41.916185 5850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830: No bootstrap required, opened a new log
I20260806 10:52:41.923563 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:41.917114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:44741 (local address 127.0.0.1:46083)
0806 10:52:41.917302 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:41.917307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:41.917729 (+ 422us) server_negotiation.cc:407] Connection header received
0806 10:52:41.917861 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:41.917867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:41.917939 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:41.918047 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:41.918498 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:41.919598 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:41.920853 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:41.921560 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:41.921723 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:41.922779 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:41.922785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:41.922790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:41.922808 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:41.922849 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:41.922868 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:41.922980 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:41.923161 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 10:52:41.923232 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:52:41.925659 5850 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a3cd09e67a46a284d228d940060830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46083 } }
I20260806 10:52:41.926003 5850 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:41.926111 5850 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3cd09e67a46a284d228d940060830, State: Initialized, Role: FOLLOWER
I20260806 10:52:41.926661 5850 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a3cd09e67a46a284d228d940060830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46083 } }
I20260806 10:52:41.926905 5850 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:41.927058 5850 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:41.927212 5850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:41.929451 5850 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a3cd09e67a46a284d228d940060830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46083 } }
I20260806 10:52:41.929937 5850 leader_election.cc:302] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a3cd09e67a46a284d228d940060830; no voters:
I20260806 10:52:41.930503 5850 leader_election.cc:288] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:41.930617 5860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:41.931347 5860 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3cd09e67a46a284d228d940060830, State: Running, Role: LEADER
I20260806 10:52:41.931881 5860 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a3cd09e67a46a284d228d940060830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46083 } }
I20260806 10:52:41.932492 5850 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:41.934947 5862 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a3cd09e67a46a284d228d940060830. Latest consensus state: current_term: 1 leader_uuid: "a2a3cd09e67a46a284d228d940060830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a3cd09e67a46a284d228d940060830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46083 } } }
I20260806 10:52:41.935297 5862 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:41.934859 5861 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a3cd09e67a46a284d228d940060830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a3cd09e67a46a284d228d940060830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46083 } } }
I20260806 10:52:41.935806 5861 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:41.940963 5868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:41.945617 5868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:41.953688 5868 catalog_manager.cc:1347] Generated new cluster ID: 236530bca8164b01a298a8e03070a1b8
I20260806 10:52:41.953794 5868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:41.968917 5868 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:41.970062 5868 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:41.979442 5868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a3cd09e67a46a284d228d940060830: Generated new TSK 0
I20260806 10:52:41.980356 5868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:41.992563 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5781
W20260806 10:52:41.993634 5845 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.101:56535
W20260806 10:52:41.994294 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5781: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:41.994437 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5781
W20260806 10:52:42.014048 5766 connection.cc:535] client connection to 127.0.0.1:46083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46083): Connection reset by peer (error 104)
2026-08-06T10:52:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:42 dist-test-slave-0b58 krb5kdc[5882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:42 dist-test-slave-0b58 krb5kdc[5882](info): set up 2 sockets
Aug 06 10:52:42 dist-test-slave-0b58 krb5kdc[5882](info): commencing operation
krb5kdc: starting...
W20260806 10:52:43.128633 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.006s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:43 dist-test-slave-0b58 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:52:43.435153 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:43.753383 5895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:43.753700 5895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:43.753872 5895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:43.760505 5895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:43.760644 5895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:43.760741 5895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:43.760844 5895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:43.760916 5895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:43.770038 5895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:43.771134 5895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:43.772416 5895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:43.783640 5900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:43.783733 5901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:43.783687 5903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:43.784564 5895 server_base.cc:1034] running on GCE node
I20260806 10:52:43.785238 5895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:43.786454 5895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:43.787649 5895 hybrid_clock.cc:648] HybridClock initialized: now 1786013563787619 us; error 26 us; skew 500 ppm
I20260806 10:52:43.787931 5895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:43 dist-test-slave-0b58 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:43.790863 5895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:52:43.792198 5895 webserver.cc:460] Webserver started at http://127.0.0.1:42155/ using document root <none> and password file <none>
I20260806 10:52:43.792866 5895 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:43.792964 5895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:43.793202 5895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:43.795964 5895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "b8c56a653de3406d8d83c60e85ed9b59"
format_stamp: "Formatted at 2026-08-06 10:52:43 on dist-test-slave-0b58"
server_key: "9450a5fe7572b6a2bf22e27383f3aa6b"
server_key_iv: "94028221afeda734bed3abd891f80e1f"
server_key_version: "encryptionkey@0"
I20260806 10:52:43.796701 5895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "b8c56a653de3406d8d83c60e85ed9b59"
format_stamp: "Formatted at 2026-08-06 10:52:43 on dist-test-slave-0b58"
server_key: "9450a5fe7572b6a2bf22e27383f3aa6b"
server_key_iv: "94028221afeda734bed3abd891f80e1f"
server_key_version: "encryptionkey@0"
I20260806 10:52:43.801486 5895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 10:52:43.804641 5911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:43.805907 5895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 10:52:43.806042 5895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "b8c56a653de3406d8d83c60e85ed9b59"
format_stamp: "Formatted at 2026-08-06 10:52:43 on dist-test-slave-0b58"
server_key: "9450a5fe7572b6a2bf22e27383f3aa6b"
server_key_iv: "94028221afeda734bed3abd891f80e1f"
server_key_version: "encryptionkey@0"
I20260806 10:52:43.806217 5895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:43.811672 5895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:43.815578 5895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:43.815816 5895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:43.834654 5895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43901
I20260806 10:52:43.834697 5964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43901 every 8 connection(s)
I20260806 10:52:43.835996 5895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:43.838073 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 5895
I20260806 10:52:43.838259 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:43.838874 16468 external_mini_cluster.cc:1428] Setting key be7a8fd45f589c889508c859a9d98041
I20260806 10:52:43.840315 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:43.846338 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:43.840680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:34285 (local address 127.0.0.1:43901)
0806 10:52:43.841463 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:43.841474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:43.841515 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:52:43.843058 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:43.843077 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:43.843406 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:43.843725 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:43.844787 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:43.844793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:43.844808 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:43.844822 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:43.844910 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:43.844928 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:43.845088 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:43.845222 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 10:52:43.845424 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":370,"threads_started":1}
I20260806 10:52:43.849920 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:43.853097 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59: Bootstrap starting.
I20260806 10:52:43.856851 5965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:43.856874 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:43.854483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:36987 (local address 127.0.0.1:43901)
0806 10:52:43.854680 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:43.854687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:43.855025 (+ 338us) server_negotiation.cc:407] Connection header received
0806 10:52:43.855181 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:43.855184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:43.855224 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:43.855300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:43.856187 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:43.856191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:43.856194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:43.856207 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:43.856241 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:43.856305 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:43.856397 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:43.856539 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:52:43.856581 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:52:43.858160 5965 log.cc:824] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:43.860729 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59: No bootstrap required, opened a new log
I20260806 10:52:43.867611 5965 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c56a653de3406d8d83c60e85ed9b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260806 10:52:43.867966 5965 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:43.868072 5965 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c56a653de3406d8d83c60e85ed9b59, State: Initialized, Role: FOLLOWER
I20260806 10:52:43.868633 5965 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c56a653de3406d8d83c60e85ed9b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260806 10:52:43.868883 5965 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:43.869019 5965 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:43.869190 5965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:43.871070 5965 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c56a653de3406d8d83c60e85ed9b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260806 10:52:43.871529 5965 leader_election.cc:302] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c56a653de3406d8d83c60e85ed9b59; no voters:
I20260806 10:52:43.872069 5965 leader_election.cc:288] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:43.872172 5975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:43.872915 5975 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [term 1 LEADER]: Becoming Leader. State: Replica: b8c56a653de3406d8d83c60e85ed9b59, State: Running, Role: LEADER
I20260806 10:52:43.873494 5975 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c56a653de3406d8d83c60e85ed9b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260806 10:52:43.874111 5965 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:43.876127 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c56a653de3406d8d83c60e85ed9b59. Latest consensus state: current_term: 1 leader_uuid: "b8c56a653de3406d8d83c60e85ed9b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c56a653de3406d8d83c60e85ed9b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } } }
I20260806 10:52:43.876716 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:43.876660 5976 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c56a653de3406d8d83c60e85ed9b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c56a653de3406d8d83c60e85ed9b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } } }
I20260806 10:52:43.877043 5976 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:43.878926 5983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:43.884595 5983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:43.893122 5983 catalog_manager.cc:1347] Generated new cluster ID: bddb11ec0e9743db9784efaaed1bb4e0
I20260806 10:52:43.893247 5983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:43.914091 5983 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:43.914491 5983 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:43.924850 5983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c56a653de3406d8d83c60e85ed9b59: Generated new TSK 0
I20260806 10:52:43.925630 5983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:43.940974 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5895
W20260806 10:52:43.942270 5960 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.101:34285
W20260806 10:52:43.943460 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 5895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:43.943712 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 5895
2026-08-06T10:52:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:44 dist-test-slave-0b58 krb5kdc[5996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:44 dist-test-slave-0b58 krb5kdc[5996](info): set up 2 sockets
Aug 06 10:52:44 dist-test-slave-0b58 krb5kdc[5996](info): commencing operation
krb5kdc: starting...
W20260806 10:52:45.100090 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.005s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:45 dist-test-slave-0b58 krb5kdc[5996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:45.397936 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35277
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:45.728322 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:45.728602 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:45.728751 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:45.735836 6009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:45.735935 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:45.736078 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:45.736181 6009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:45.736289 6009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:45.745587 6009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:45.746827 6009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:45.748093 6009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:45.759982 6017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:45.760063 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:45.760038 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:45.760982 6009 server_base.cc:1034] running on GCE node
I20260806 10:52:45.761727 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:45.763003 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:45.764207 6009 hybrid_clock.cc:648] HybridClock initialized: now 1786013565764173 us; error 43 us; skew 500 ppm
I20260806 10:52:45.764559 6009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:45 dist-test-slave-0b58 krb5kdc[5996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013565, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:45.767616 6009 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:45.769054 6009 webserver.cc:460] Webserver started at http://127.16.21.62:43165/ using document root <none> and password file <none>
I20260806 10:52:45.769701 6009 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:45.769853 6009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:45.770150 6009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:45.773489 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "46102727757440abb15cceab458e9b04"
format_stamp: "Formatted at 2026-08-06 10:52:45 on dist-test-slave-0b58"
server_key: "311d4bd3d66198d1356925f0011eabe7"
server_key_iv: "1b820292980de6aead8efa5487bd1b67"
server_key_version: "encryptionkey@0"
I20260806 10:52:45.774328 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "46102727757440abb15cceab458e9b04"
format_stamp: "Formatted at 2026-08-06 10:52:45 on dist-test-slave-0b58"
server_key: "311d4bd3d66198d1356925f0011eabe7"
server_key_iv: "1b820292980de6aead8efa5487bd1b67"
server_key_version: "encryptionkey@0"
I20260806 10:52:45.779508 6009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:52:45.782964 6024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:45.784390 6009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:45.784540 6009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "46102727757440abb15cceab458e9b04"
format_stamp: "Formatted at 2026-08-06 10:52:45 on dist-test-slave-0b58"
server_key: "311d4bd3d66198d1356925f0011eabe7"
server_key_iv: "1b820292980de6aead8efa5487bd1b67"
server_key_version: "encryptionkey@0"
I20260806 10:52:45.784700 6009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:45.798050 6009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:45.801860 6009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:45.802088 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:45.820724 6009 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35277
I20260806 10:52:45.820761 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35277 every 8 connection(s)
I20260806 10:52:45.822049 6009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:45.825692 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:45.827487 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6009
I20260806 10:52:45.827672 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:45.828109 16468 external_mini_cluster.cc:1428] Setting key 1b3761f9fc4bb2fb1f430fda2b3481cd
I20260806 10:52:45.839927 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04: Bootstrap starting.
Aug 06 10:52:45 dist-test-slave-0b58 krb5kdc[5996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:45.843181 6077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:45.844465 6077 log.cc:824] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:45.847225 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04: No bootstrap required, opened a new log
I20260806 10:52:45.851512 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:45.829992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50693 (local address 127.16.21.62:35277)
0806 10:52:45.830851 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:45.830866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:45.830928 (+ 62us) server_negotiation.cc:407] Connection header received
0806 10:52:45.832640 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:45.832668 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:45.833167 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:45.833580 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:45.834865 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:45.836982 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:45.838279 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:45.839240 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:45.839410 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:45.843977 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:45.843984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:45.843997 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:45.844030 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:45.847190 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:45.848146 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:45.848152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:45.848157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:45.848388 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:45.848892 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:45.848895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:45.848897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:45.849397 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:45.849631 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:45.850154 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 10:52:45.850476 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":360,"threads_started":1}
I20260806 10:52:45.855542 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:45.856631 6077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46102727757440abb15cceab458e9b04" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35277 } }
I20260806 10:52:45.856953 6077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:45.857044 6077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46102727757440abb15cceab458e9b04, State: Initialized, Role: FOLLOWER
I20260806 10:52:45.857570 6077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46102727757440abb15cceab458e9b04" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35277 } }
I20260806 10:52:45.857788 6077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:45.857936 6077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:45.858098 6077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:45.860127 6077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46102727757440abb15cceab458e9b04" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35277 } }
I20260806 10:52:45.860654 6077 leader_election.cc:302] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46102727757440abb15cceab458e9b04; no voters:
I20260806 10:52:45.861370 6077 leader_election.cc:288] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:45.861515 6085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:45.863523 6077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:45.864732 6085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [term 1 LEADER]: Becoming Leader. State: Replica: 46102727757440abb15cceab458e9b04, State: Running, Role: LEADER
I20260806 10:52:45.865263 6085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46102727757440abb15cceab458e9b04" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35277 } }
I20260806 10:52:45.868225 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46102727757440abb15cceab458e9b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46102727757440abb15cceab458e9b04" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35277 } } }
I20260806 10:52:45.868706 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:45.868371 6088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46102727757440abb15cceab458e9b04. Latest consensus state: current_term: 1 leader_uuid: "46102727757440abb15cceab458e9b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46102727757440abb15cceab458e9b04" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35277 } } }
I20260806 10:52:45.869146 6088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:45.873222 6097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:45.878599 6097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:45.884032 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:45.863462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46933 (local address 127.16.21.62:35277)
0806 10:52:45.863929 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:45.863937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:45.863961 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:52:45.864024 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:45.864035 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:45.864124 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:45.864283 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:45.866461 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:45.867889 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:45.869907 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:45.870740 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:45.870894 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:45.874620 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:45.874628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:45.874638 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:45.874678 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:45.878586 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:45.879888 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:45.879896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:45.879903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:45.880128 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:45.880919 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:45.880927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:45.880932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:45.881514 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:45.883025 (+ 1511us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:45.883259 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 10:52:45.883501 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260806 10:52:45.887976 6097 catalog_manager.cc:1347] Generated new cluster ID: c3bd7ac6129a439fb2b07bb434eb33c5
I20260806 10:52:45.888083 6097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:45.903214 6097 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:45.904660 6097 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:45.915055 6097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46102727757440abb15cceab458e9b04: Generated new TSK 0
I20260806 10:52:45.916050 6097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:45.922921 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6009
I20260806 10:52:45.924041 6072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:46.014371 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6009
W20260806 10:52:46.034525 5994 connection.cc:535] client connection to 127.16.21.62:35277 recv error: Network error: failed to read from TLS socket (remote: 127.16.21.62:35277): Connection reset by peer (error 104)
2026-08-06T10:52:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2070 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:46 dist-test-slave-0b58 krb5kdc[6110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:46 dist-test-slave-0b58 krb5kdc[6110](info): set up 2 sockets
Aug 06 10:52:46 dist-test-slave-0b58 krb5kdc[6110](info): commencing operation
krb5kdc: starting...
W20260806 10:52:47.151973 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:47 dist-test-slave-0b58 krb5kdc[6110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:47.453334 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38493
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:47.782033 6123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:47.782290 6123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:47.782377 6123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:47.789371 6123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:47.789489 6123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:47.789530 6123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:47.789587 6123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:47.789635 6123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:47.798614 6123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:38493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:47.799676 6123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:47.800876 6123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:47.812435 6128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:47.812482 6131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:47.812508 6129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:47.813362 6123 server_base.cc:1034] running on GCE node
I20260806 10:52:47.814087 6123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:47.815342 6123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:47.816565 6123 hybrid_clock.cc:648] HybridClock initialized: now 1786013567816523 us; error 54 us; skew 500 ppm
I20260806 10:52:47.816905 6123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:47 dist-test-slave-0b58 krb5kdc[6110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013567, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:47.823123 6123 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:47.824503 6123 webserver.cc:460] Webserver started at http://127.16.21.62:43923/ using document root <none> and password file <none>
I20260806 10:52:47.825088 6123 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:47.825187 6123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:47.825425 6123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:47.828217 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "b219abe30ff04a9aa665696042481023"
format_stamp: "Formatted at 2026-08-06 10:52:47 on dist-test-slave-0b58"
server_key: "62d0d543827c7c8f8d775d244aaabd4b"
server_key_iv: "df5ee9d62ba28d50435ebf7bb14f7903"
server_key_version: "encryptionkey@0"
I20260806 10:52:47.828982 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "b219abe30ff04a9aa665696042481023"
format_stamp: "Formatted at 2026-08-06 10:52:47 on dist-test-slave-0b58"
server_key: "62d0d543827c7c8f8d775d244aaabd4b"
server_key_iv: "df5ee9d62ba28d50435ebf7bb14f7903"
server_key_version: "encryptionkey@0"
I20260806 10:52:47.833802 6123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:52:47.836864 6138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:47.837998 6123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:52:47.838117 6123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "b219abe30ff04a9aa665696042481023"
format_stamp: "Formatted at 2026-08-06 10:52:47 on dist-test-slave-0b58"
server_key: "62d0d543827c7c8f8d775d244aaabd4b"
server_key_iv: "df5ee9d62ba28d50435ebf7bb14f7903"
server_key_version: "encryptionkey@0"
I20260806 10:52:47.838258 6123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:47.853114 6123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:47.857976 6123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:47.858196 6123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:47.876681 6123 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:38493
I20260806 10:52:47.876765 6190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:38493 every 8 connection(s)
I20260806 10:52:47.877993 6123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:47.881537 6191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:47.882710 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6123
I20260806 10:52:47.882912 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:47.883478 16468 external_mini_cluster.cc:1428] Setting key 48faff69a85656a5a75d770e60809761
I20260806 10:52:47.893187 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023: Bootstrap starting.
I20260806 10:52:47.895509 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:47.885259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54165 (local address 127.16.21.62:38493)
0806 10:52:47.885977 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:47.885990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:47.886027 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:52:47.887637 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:47.887656 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:47.887991 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:47.888394 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:47.888866 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:47.890047 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:47.891282 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:47.891881 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:47.892068 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:47.893395 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:47.893403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:47.893415 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:47.893946 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:47.894017 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:47.894041 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:47.894205 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:47.894528 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 10:52:47.894668 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":305,"threads_started":1}
I20260806 10:52:47.896240 6191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:47.897297 6191 log.cc:824] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:47.899132 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:47.900358 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023: No bootstrap required, opened a new log
I20260806 10:52:47.910210 6191 raft_consensus.cc:348] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b219abe30ff04a9aa665696042481023" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38493 } }
I20260806 10:52:47.910651 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:47.904197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35075 (local address 127.16.21.62:38493)
0806 10:52:47.904405 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:47.904414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:47.904867 (+ 453us) server_negotiation.cc:407] Connection header received
0806 10:52:47.904999 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:47.905004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:47.905071 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:47.905172 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:47.905671 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:47.906863 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:47.908151 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:47.908877 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:47.909006 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:47.909972 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:47.909978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:47.909982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:47.910000 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:47.910042 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:47.910060 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:47.910167 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:47.910318 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 10:52:47.910364 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:52:47.910660 6191 raft_consensus.cc:374] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:47.910831 6191 raft_consensus.cc:729] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b219abe30ff04a9aa665696042481023, State: Initialized, Role: FOLLOWER
I20260806 10:52:47.911412 6191 consensus_queue.cc:260] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b219abe30ff04a9aa665696042481023" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38493 } }
I20260806 10:52:47.911656 6191 raft_consensus.cc:388] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:47.911819 6191 raft_consensus.cc:482] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:47.911983 6191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:47.914229 6191 raft_consensus.cc:504] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b219abe30ff04a9aa665696042481023" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38493 } }
I20260806 10:52:47.914697 6191 leader_election.cc:302] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b219abe30ff04a9aa665696042481023; no voters:
I20260806 10:52:47.915246 6191 leader_election.cc:288] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:47.915395 6201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:47.916756 6201 raft_consensus.cc:686] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [term 1 LEADER]: Becoming Leader. State: Replica: b219abe30ff04a9aa665696042481023, State: Running, Role: LEADER
I20260806 10:52:47.917258 6191 sys_catalog.cc:564] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:47.917263 6201 consensus_queue.cc:237] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b219abe30ff04a9aa665696042481023" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38493 } }
I20260806 10:52:47.919924 6203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b219abe30ff04a9aa665696042481023. Latest consensus state: current_term: 1 leader_uuid: "b219abe30ff04a9aa665696042481023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b219abe30ff04a9aa665696042481023" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38493 } } }
I20260806 10:52:47.920221 6203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:47.920524 6202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b219abe30ff04a9aa665696042481023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b219abe30ff04a9aa665696042481023" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38493 } } }
I20260806 10:52:47.920850 6202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:47.922264 6207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:47.927238 6207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:47.936139 6207 catalog_manager.cc:1347] Generated new cluster ID: 6124500e0dde4e2083a232825f56b163
I20260806 10:52:47.936316 6207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:47.949450 6207 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:47.950529 6207 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:47.963636 6207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b219abe30ff04a9aa665696042481023: Generated new TSK 0
I20260806 10:52:47.964505 6207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:47.980618 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6123
W20260806 10:52:47.981676 6186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54165
W20260806 10:52:47.982365 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:47.982508 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6123
W20260806 10:52:48.002112 6108 connection.cc:535] client connection to 127.16.21.62:38493 recv error: Network error: failed to read from TLS socket (remote: 127.16.21.62:38493): Connection reset by peer (error 104)
2026-08-06T10:52:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:48 dist-test-slave-0b58 krb5kdc[6222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:48 dist-test-slave-0b58 krb5kdc[6222](info): set up 2 sockets
Aug 06 10:52:48 dist-test-slave-0b58 krb5kdc[6222](info): commencing operation
krb5kdc: starting...
W20260806 10:52:49.087622 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:49 dist-test-slave-0b58 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:49.389454 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35885
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:49.711285 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:49.711604 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:49.711784 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:49.718784 6235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:49.718938 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:49.719038 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:49.719149 6235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:49.719230 6235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:49.728924 6235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:49.730119 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:49.731359 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:49.742870 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:49.742931 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:49.742969 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:49.743925 6235 server_base.cc:1034] running on GCE node
I20260806 10:52:49.744684 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:49.745939 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:49.747140 6235 hybrid_clock.cc:648] HybridClock initialized: now 1786013569747092 us; error 39 us; skew 500 ppm
I20260806 10:52:49.747481 6235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:49 dist-test-slave-0b58 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:49.750454 6235 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:49.751920 6235 webserver.cc:460] Webserver started at http://127.16.21.62:45733/ using document root <none> and password file <none>
I20260806 10:52:49.752652 6235 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:49.752803 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:49.753110 6235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:49.755942 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "ec0721f6fb41411b96e4e2413c4df73c"
format_stamp: "Formatted at 2026-08-06 10:52:49 on dist-test-slave-0b58"
server_key: "cfef8ab39889d4b51c8836b725b06e03"
server_key_iv: "b7f881a7f4c2263f3cd31906101b54f2"
server_key_version: "encryptionkey@0"
I20260806 10:52:49.756753 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "ec0721f6fb41411b96e4e2413c4df73c"
format_stamp: "Formatted at 2026-08-06 10:52:49 on dist-test-slave-0b58"
server_key: "cfef8ab39889d4b51c8836b725b06e03"
server_key_iv: "b7f881a7f4c2263f3cd31906101b54f2"
server_key_version: "encryptionkey@0"
I20260806 10:52:49.761538 6235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 10:52:49.764746 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:49.765947 6235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 10:52:49.766117 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "ec0721f6fb41411b96e4e2413c4df73c"
format_stamp: "Formatted at 2026-08-06 10:52:49 on dist-test-slave-0b58"
server_key: "cfef8ab39889d4b51c8836b725b06e03"
server_key_iv: "b7f881a7f4c2263f3cd31906101b54f2"
server_key_version: "encryptionkey@0"
I20260806 10:52:49.766321 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:49.771647 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:49.775231 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:49.775506 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:49.793869 6235 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35885
I20260806 10:52:49.793918 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35885 every 8 connection(s)
I20260806 10:52:49.795218 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:49.797427 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6235
I20260806 10:52:49.797616 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:49.798159 16468 external_mini_cluster.cc:1428] Setting key e5c5a099b2a3fe9f36a21c9d0f9a4429
I20260806 10:52:49.799161 6302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:49.805539 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:49.799914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41663 (local address 127.16.21.62:35885)
0806 10:52:49.800728 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:49.800739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:49.800776 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:52:49.802240 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:49.802259 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:49.802575 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:49.802975 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:49.804021 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:49.804027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:49.804038 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:49.804053 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:49.804119 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:49.804138 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:49.804405 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:49.804543 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 10:52:49.804703 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":385,"threads_started":1}
I20260806 10:52:49.808888 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:49.811004 6302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c: Bootstrap starting.
I20260806 10:52:49.814214 6302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:49.815449 6302 log.cc:824] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:49.815522 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:49.812997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55985 (local address 127.16.21.62:35885)
0806 10:52:49.813162 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:49.813169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:49.813734 (+ 565us) server_negotiation.cc:407] Connection header received
0806 10:52:49.813853 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:49.813857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:49.813927 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:49.814041 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:49.814921 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:49.814926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:49.814930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:49.814946 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:49.814988 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:49.815025 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:49.815155 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:49.815208 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 10:52:49.815267 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:52:49.818053 6302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c: No bootstrap required, opened a new log
I20260806 10:52:49.825299 6302 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0721f6fb41411b96e4e2413c4df73c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35885 } }
I20260806 10:52:49.825613 6302 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:49.825718 6302 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0721f6fb41411b96e4e2413c4df73c, State: Initialized, Role: FOLLOWER
I20260806 10:52:49.826239 6302 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0721f6fb41411b96e4e2413c4df73c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35885 } }
I20260806 10:52:49.826460 6302 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:49.826601 6302 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:49.826759 6302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:49.828810 6302 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0721f6fb41411b96e4e2413c4df73c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35885 } }
I20260806 10:52:49.829308 6302 leader_election.cc:302] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0721f6fb41411b96e4e2413c4df73c; no voters:
I20260806 10:52:49.829905 6302 leader_election.cc:288] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:49.830065 6313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:49.831037 6313 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [term 1 LEADER]: Becoming Leader. State: Replica: ec0721f6fb41411b96e4e2413c4df73c, State: Running, Role: LEADER
I20260806 10:52:49.831604 6313 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0721f6fb41411b96e4e2413c4df73c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35885 } }
I20260806 10:52:49.832003 6302 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:49.834435 6315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0721f6fb41411b96e4e2413c4df73c. Latest consensus state: current_term: 1 leader_uuid: "ec0721f6fb41411b96e4e2413c4df73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0721f6fb41411b96e4e2413c4df73c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35885 } } }
I20260806 10:52:49.834829 6315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:49.836576 6314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0721f6fb41411b96e4e2413c4df73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0721f6fb41411b96e4e2413c4df73c" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35885 } } }
I20260806 10:52:49.836809 6320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:49.836836 6314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:49.842015 6320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:49.850538 6320 catalog_manager.cc:1347] Generated new cluster ID: 659995b8d8e840bdbc88e80a9d832dce
I20260806 10:52:49.850638 6320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:49.861402 6320 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:49.861809 6320 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:49.877151 6320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0721f6fb41411b96e4e2413c4df73c: Generated new TSK 0
I20260806 10:52:49.877991 6320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:49.890162 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6235
W20260806 10:52:49.891544 6297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41663
W20260806 10:52:49.892145 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6235: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:49.892334 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6235
2026-08-06T10:52:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:50 dist-test-slave-0b58 krb5kdc[6335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:50 dist-test-slave-0b58 krb5kdc[6335](info): set up 2 sockets
Aug 06 10:52:50 dist-test-slave-0b58 krb5kdc[6335](info): commencing operation
krb5kdc: starting...
W20260806 10:52:50.967973 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:51 dist-test-slave-0b58 krb5kdc[6335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:51.257169 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35243
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:51.572721 6348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:51.573005 6348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:51.573160 6348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:51.579988 6348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:51.580078 6348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:51.580207 6348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:51.580302 6348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:51.580384 6348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:51.589788 6348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:51.590910 6348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:51.592106 6348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:51.603296 6356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:51.603344 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:51.603374 6354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:51.604243 6348 server_base.cc:1034] running on GCE node
I20260806 10:52:51.605015 6348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:51.606189 6348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:51.607446 6348 hybrid_clock.cc:648] HybridClock initialized: now 1786013571607398 us; error 33 us; skew 500 ppm
I20260806 10:52:51.607753 6348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:51 dist-test-slave-0b58 krb5kdc[6335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:51.610620 6348 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:51.612031 6348 webserver.cc:460] Webserver started at http://127.16.21.62:38691/ using document root <none> and password file <none>
I20260806 10:52:51.612701 6348 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:51.612829 6348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:51.613168 6348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:51.615849 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "d55f2efae40a46dd8aa1ce52929ba429"
format_stamp: "Formatted at 2026-08-06 10:52:51 on dist-test-slave-0b58"
server_key: "e51c2a5ad045ac011f43f963dc1fe51e"
server_key_iv: "8635d65bf0a46b36741fcc24a816dec3"
server_key_version: "encryptionkey@0"
I20260806 10:52:51.616641 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "d55f2efae40a46dd8aa1ce52929ba429"
format_stamp: "Formatted at 2026-08-06 10:52:51 on dist-test-slave-0b58"
server_key: "e51c2a5ad045ac011f43f963dc1fe51e"
server_key_iv: "8635d65bf0a46b36741fcc24a816dec3"
server_key_version: "encryptionkey@0"
I20260806 10:52:51.621309 6348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:52:51.624522 6363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:51.625619 6348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 10:52:51.625787 6348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "d55f2efae40a46dd8aa1ce52929ba429"
format_stamp: "Formatted at 2026-08-06 10:52:51 on dist-test-slave-0b58"
server_key: "e51c2a5ad045ac011f43f963dc1fe51e"
server_key_iv: "8635d65bf0a46b36741fcc24a816dec3"
server_key_version: "encryptionkey@0"
I20260806 10:52:51.625967 6348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:51.654310 6348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:51.658376 6348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:51.658634 6348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:51.676529 6348 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:35243
I20260806 10:52:51.676597 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:35243 every 8 connection(s)
I20260806 10:52:51.677698 6348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:51.681005 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:51.681427 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6348
I20260806 10:52:51.681622 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:51.682163 16468 external_mini_cluster.cc:1428] Setting key cf360070fa6f862b3569d349f635cf34
I20260806 10:52:51.694084 6415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429: Bootstrap starting.
Aug 06 10:52:51 dist-test-slave-0b58 krb5kdc[6335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:51.697350 6415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:51.698339 6415 log.cc:824] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:51.700958 6415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429: No bootstrap required, opened a new log
I20260806 10:52:51.704174 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:51.683830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58355 (local address 127.16.21.62:35243)
0806 10:52:51.684670 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:51.684679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:51.684714 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:52:51.686912 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:51.686931 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:51.687276 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:51.687591 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:51.689182 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:51.690284 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:51.691901 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:51.692558 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:51.692667 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:51.697124 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:51.697131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:51.697142 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:51.697173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:51.699924 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:51.700615 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:51.700622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:51.700627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:51.700777 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:51.701157 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:51.701162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:51.701165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:51.701883 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:51.702175 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:51.702461 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 10:52:51.702868 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":456,"threads_started":1}
I20260806 10:52:51.707585 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:51.710436 6415 raft_consensus.cc:348] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f2efae40a46dd8aa1ce52929ba429" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35243 } }
I20260806 10:52:51.710851 6415 raft_consensus.cc:374] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:51.710975 6415 raft_consensus.cc:729] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f2efae40a46dd8aa1ce52929ba429, State: Initialized, Role: FOLLOWER
I20260806 10:52:51.711618 6415 consensus_queue.cc:260] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f2efae40a46dd8aa1ce52929ba429" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35243 } }
I20260806 10:52:51.711912 6415 raft_consensus.cc:388] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:51.712051 6415 raft_consensus.cc:482] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:51.712205 6415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:51.714426 6415 raft_consensus.cc:504] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f2efae40a46dd8aa1ce52929ba429" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35243 } }
I20260806 10:52:51.714970 6415 leader_election.cc:302] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d55f2efae40a46dd8aa1ce52929ba429; no voters:
I20260806 10:52:51.715705 6415 leader_election.cc:288] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:51.715813 6425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:51.717386 6425 raft_consensus.cc:686] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [term 1 LEADER]: Becoming Leader. State: Replica: d55f2efae40a46dd8aa1ce52929ba429, State: Running, Role: LEADER
I20260806 10:52:51.717967 6415 sys_catalog.cc:564] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:51.717883 6425 consensus_queue.cc:237] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f2efae40a46dd8aa1ce52929ba429" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35243 } }
I20260806 10:52:51.725781 6426 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d55f2efae40a46dd8aa1ce52929ba429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f2efae40a46dd8aa1ce52929ba429" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35243 } } }
I20260806 10:52:51.726333 6426 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:51.727048 6427 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d55f2efae40a46dd8aa1ce52929ba429. Latest consensus state: current_term: 1 leader_uuid: "d55f2efae40a46dd8aa1ce52929ba429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f2efae40a46dd8aa1ce52929ba429" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 35243 } } }
I20260806 10:52:51.727437 6427 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:51.730132 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:51.712764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45669 (local address 127.16.21.62:35243)
0806 10:52:51.712953 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:51.712959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:51.715148 (+ 2189us) server_negotiation.cc:407] Connection header received
0806 10:52:51.715294 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:51.715297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:51.715349 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:51.715460 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:51.717152 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:51.718026 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:51.719367 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:51.719750 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:51.719883 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:51.722149 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:51.722154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:51.722157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:51.722183 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:51.725520 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:51.726443 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:51.726449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:51.726452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:51.726595 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:51.727035 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:51.727039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:51.727042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:51.727554 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:51.727877 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:51.729273 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0806 10:52:51.729528 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:52:51.732738 6431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:51.737830 6431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:51.746740 6431 catalog_manager.cc:1347] Generated new cluster ID: f02fa3fd80794f0db920eb313662abb1
I20260806 10:52:51.746862 6431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:51.778195 6431 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:51.779765 6431 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:51.791970 6431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d55f2efae40a46dd8aa1ce52929ba429: Generated new TSK 0
I20260806 10:52:51.792695 6431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:51.806728 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6348
I20260806 10:52:51.807806 6410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:52:51.897617 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6348
2026-08-06T10:52:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:52 dist-test-slave-0b58 krb5kdc[6448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:52 dist-test-slave-0b58 krb5kdc[6448](info): set up 2 sockets
Aug 06 10:52:52 dist-test-slave-0b58 krb5kdc[6448](info): commencing operation
krb5kdc: starting...
W20260806 10:52:52.979687 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:53 dist-test-slave-0b58 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:53.261502 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36529
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:53.573993 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:53.574313 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:53.574467 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:53.581073 6461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:53.581200 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:53.581300 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:53.581400 6461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:53.581480 6461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:53.590488 6461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:36529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:53.591564 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:53.592725 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:53.603717 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:53.603788 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:53.603825 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:53.604882 6461 server_base.cc:1034] running on GCE node
I20260806 10:52:53.605594 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:53.606819 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:53.608033 6461 hybrid_clock.cc:648] HybridClock initialized: now 1786013573607979 us; error 40 us; skew 500 ppm
I20260806 10:52:53.608433 6461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:53 dist-test-slave-0b58 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013573, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:53.615864 6461 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:53.617244 6461 webserver.cc:460] Webserver started at http://127.16.21.62:34779/ using document root <none> and password file <none>
I20260806 10:52:53.617873 6461 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:53.618000 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:53.618292 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:53.621025 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "b857bf3a85164bb5b6f6e7a8268b10a3"
format_stamp: "Formatted at 2026-08-06 10:52:53 on dist-test-slave-0b58"
server_key: "408cb2c66372bebe8521227f92a107c9"
server_key_iv: "26d2d838d3ccdbe7aab0cc3ce62df23e"
server_key_version: "encryptionkey@0"
I20260806 10:52:53.621738 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "b857bf3a85164bb5b6f6e7a8268b10a3"
format_stamp: "Formatted at 2026-08-06 10:52:53 on dist-test-slave-0b58"
server_key: "408cb2c66372bebe8521227f92a107c9"
server_key_iv: "26d2d838d3ccdbe7aab0cc3ce62df23e"
server_key_version: "encryptionkey@0"
I20260806 10:52:53.626472 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 10:52:53.629585 6476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:53.630764 6461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 10:52:53.630932 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "b857bf3a85164bb5b6f6e7a8268b10a3"
format_stamp: "Formatted at 2026-08-06 10:52:53 on dist-test-slave-0b58"
server_key: "408cb2c66372bebe8521227f92a107c9"
server_key_iv: "26d2d838d3ccdbe7aab0cc3ce62df23e"
server_key_version: "encryptionkey@0"
I20260806 10:52:53.631114 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:53.649744 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:53.653266 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:53.653537 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:53.671950 6461 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:36529
I20260806 10:52:53.671995 6527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:36529 every 8 connection(s)
I20260806 10:52:53.673230 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:53.675897 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6461
I20260806 10:52:53.676079 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:53.676644 16468 external_mini_cluster.cc:1428] Setting key 6aa698ec49589494af0b0855b88b2de3
I20260806 10:52:53.677222 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:53.689616 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:53.678393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59151 (local address 127.16.21.62:36529)
0806 10:52:53.679135 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:53.679145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:53.679180 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:52:53.681403 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:53.681421 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:53.681734 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:53.682002 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:53.682537 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:53.683791 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:53.685510 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:53.686069 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:53.686232 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:53.687592 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:53.687599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:53.687610 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:53.688101 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:53.688173 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:53.688208 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:53.688371 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:53.688580 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 10:52:53.688822 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":376,"threads_started":1}
I20260806 10:52:53.690163 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3: Bootstrap starting.
I20260806 10:52:53.692941 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:53.693194 6528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:53.694361 6528 log.cc:824] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:53.697219 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3: No bootstrap required, opened a new log
I20260806 10:52:53.703989 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:53.697855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50267 (local address 127.16.21.62:36529)
0806 10:52:53.698028 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:53.698033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:53.698414 (+ 381us) server_negotiation.cc:407] Connection header received
0806 10:52:53.698535 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:53.698540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:53.698607 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:53.698705 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:53.699161 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:53.700293 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:53.701536 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:53.702228 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:53.702352 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:53.703279 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:53.703285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:53.703289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:53.703308 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:53.703349 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:53.703370 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:53.703468 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:53.703589 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 10:52:53.703682 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:52:53.705914 6528 raft_consensus.cc:348] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36529 } }
I20260806 10:52:53.706271 6528 raft_consensus.cc:374] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:53.706387 6528 raft_consensus.cc:729] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b857bf3a85164bb5b6f6e7a8268b10a3, State: Initialized, Role: FOLLOWER
I20260806 10:52:53.706881 6528 consensus_queue.cc:260] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36529 } }
I20260806 10:52:53.707123 6528 raft_consensus.cc:388] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:53.707276 6528 raft_consensus.cc:482] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:53.707453 6528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:53.709652 6528 raft_consensus.cc:504] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36529 } }
I20260806 10:52:53.710127 6528 leader_election.cc:302] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b857bf3a85164bb5b6f6e7a8268b10a3; no voters:
I20260806 10:52:53.710731 6528 leader_election.cc:288] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:53.710866 6538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:53.711609 6538 raft_consensus.cc:686] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [term 1 LEADER]: Becoming Leader. State: Replica: b857bf3a85164bb5b6f6e7a8268b10a3, State: Running, Role: LEADER
I20260806 10:52:53.712133 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36529 } }
I20260806 10:52:53.712785 6528 sys_catalog.cc:564] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:53.716071 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36529 } } }
I20260806 10:52:53.716482 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:53.716454 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b857bf3a85164bb5b6f6e7a8268b10a3. Latest consensus state: current_term: 1 leader_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b857bf3a85164bb5b6f6e7a8268b10a3" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 36529 } } }
I20260806 10:52:53.716760 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3 [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:53.723677 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:53.729641 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:53.737345 6545 catalog_manager.cc:1347] Generated new cluster ID: 0c39bebb0ddc4ed0be114a291a362a14
I20260806 10:52:53.737442 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:53.745206 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:53.746790 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:53.753178 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b857bf3a85164bb5b6f6e7a8268b10a3: Generated new TSK 0
I20260806 10:52:53.753822 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:53.768783 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6461
W20260806 10:52:53.769740 6523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59151
W20260806 10:52:53.770401 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:53.770568 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6461
2026-08-06T10:52:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:53 dist-test-slave-0b58 krb5kdc[6559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:53 dist-test-slave-0b58 krb5kdc[6559](info): set up 2 sockets
Aug 06 10:52:53 dist-test-slave-0b58 krb5kdc[6559](info): commencing operation
krb5kdc: starting...
W20260806 10:52:54.848021 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:55 dist-test-slave-0b58 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:55.138826 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46561
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:55.461920 6572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:55.462160 6572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:55.462267 6572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:55.469236 6572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:55.469329 6572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:55.469396 6572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:55.469455 6572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:55.469496 6572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:55.478802 6572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:55.479902 6572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:55.481108 6572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:55.491801 6577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:55.491766 6578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:55.492190 6580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:55.492606 6572 server_base.cc:1034] running on GCE node
I20260806 10:52:55.493330 6572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:55.494570 6572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:55.495760 6572 hybrid_clock.cc:648] HybridClock initialized: now 1786013575495717 us; error 37 us; skew 500 ppm
I20260806 10:52:55.496057 6572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:55 dist-test-slave-0b58 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013575, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:55.498854 6572 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:55.500212 6572 webserver.cc:460] Webserver started at http://127.16.21.62:36833/ using document root <none> and password file <none>
I20260806 10:52:55.500952 6572 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:55.501092 6572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:55.501390 6572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:55.504161 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "8a90b96872004dc38ad523d7c72f84fa"
format_stamp: "Formatted at 2026-08-06 10:52:55 on dist-test-slave-0b58"
server_key: "3de937803e586e0b6dd12f5d2bba3b13"
server_key_iv: "bc650899abbfe986fcccb74ecc45d659"
server_key_version: "encryptionkey@0"
I20260806 10:52:55.504977 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "8a90b96872004dc38ad523d7c72f84fa"
format_stamp: "Formatted at 2026-08-06 10:52:55 on dist-test-slave-0b58"
server_key: "3de937803e586e0b6dd12f5d2bba3b13"
server_key_iv: "bc650899abbfe986fcccb74ecc45d659"
server_key_version: "encryptionkey@0"
I20260806 10:52:55.509661 6572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:52:55.512693 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:55.513813 6572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 10:52:55.513988 6572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "8a90b96872004dc38ad523d7c72f84fa"
format_stamp: "Formatted at 2026-08-06 10:52:55 on dist-test-slave-0b58"
server_key: "3de937803e586e0b6dd12f5d2bba3b13"
server_key_iv: "bc650899abbfe986fcccb74ecc45d659"
server_key_version: "encryptionkey@0"
I20260806 10:52:55.514185 6572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:55.519615 6572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:55.523074 6572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:55.523375 6572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:55.540887 6572 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:46561
I20260806 10:52:55.540951 6641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:46561 every 8 connection(s)
I20260806 10:52:55.542025 6572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:55.545186 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6572
I20260806 10:52:55.545202 6642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:55.545358 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:55.545960 16468 external_mini_cluster.cc:1428] Setting key 17c31daa1472442147fb057701901139
I20260806 10:52:55.552896 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:55.547578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34163 (local address 127.16.21.62:46561)
0806 10:52:55.548435 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:55.548444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:55.548481 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:52:55.549937 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:55.549955 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:55.550235 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:55.550505 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:55.551436 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:55.551442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:55.551455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:55.551468 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:55.551527 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:55.551546 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:55.551700 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:55.551814 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 10:52:55.552006 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":424,"threads_started":1}
I20260806 10:52:55.556169 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:55.557986 6642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa: Bootstrap starting.
I20260806 10:52:55.561604 6642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:55.562659 6642 log.cc:824] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:55.563148 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:55.560868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47883 (local address 127.16.21.62:46561)
0806 10:52:55.561021 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:55.561027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:55.561523 (+ 496us) server_negotiation.cc:407] Connection header received
0806 10:52:55.561645 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:55.561650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:55.561704 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:55.561805 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:55.562533 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:55.562537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:55.562541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:52:55.562556 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:55.562597 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:52:55.562617 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:55.562733 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:55.562831 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 10:52:55.562880 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:52:55.565405 6642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa: No bootstrap required, opened a new log
I20260806 10:52:55.572535 6642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a90b96872004dc38ad523d7c72f84fa" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46561 } }
I20260806 10:52:55.572840 6642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:52:55.572945 6642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a90b96872004dc38ad523d7c72f84fa, State: Initialized, Role: FOLLOWER
I20260806 10:52:55.573513 6642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a90b96872004dc38ad523d7c72f84fa" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46561 } }
I20260806 10:52:55.573741 6642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:52:55.573885 6642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:52:55.574040 6642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:55.576020 6642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a90b96872004dc38ad523d7c72f84fa" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46561 } }
I20260806 10:52:55.576501 6642 leader_election.cc:302] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a90b96872004dc38ad523d7c72f84fa; no voters:
I20260806 10:52:55.577032 6642 leader_election.cc:288] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:52:55.577131 6653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:55.578286 6653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [term 1 LEADER]: Becoming Leader. State: Replica: 8a90b96872004dc38ad523d7c72f84fa, State: Running, Role: LEADER
I20260806 10:52:55.578852 6653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a90b96872004dc38ad523d7c72f84fa" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46561 } }
I20260806 10:52:55.579020 6642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:55.581662 6654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a90b96872004dc38ad523d7c72f84fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a90b96872004dc38ad523d7c72f84fa" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46561 } } }
I20260806 10:52:55.581682 6655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a90b96872004dc38ad523d7c72f84fa. Latest consensus state: current_term: 1 leader_uuid: "8a90b96872004dc38ad523d7c72f84fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a90b96872004dc38ad523d7c72f84fa" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 46561 } } }
I20260806 10:52:55.582129 6655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:55.582129 6654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:55.584405 6660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:55.588847 6660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:55.596326 6660 catalog_manager.cc:1347] Generated new cluster ID: 5758f6272574442e9bf297be86e09667
I20260806 10:52:55.596416 6660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:55.612052 6660 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:55.612502 6660 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:55.621237 6660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a90b96872004dc38ad523d7c72f84fa: Generated new TSK 0
I20260806 10:52:55.621994 6660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:52:55.629405 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6572
W20260806 10:52:55.630534 6637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34163
W20260806 10:52:55.631209 16468 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:52:55.631353 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6572
2026-08-06T10:52:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1861 ms)
[----------] 18 tests from AuthTokenIssuingTest (35863 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:52:55 dist-test-slave-0b58 krb5kdc[6674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:52:55 dist-test-slave-0b58 krb5kdc[6674](info): set up 2 sockets
Aug 06 10:52:55 dist-test-slave-0b58 krb5kdc[6674](info): commencing operation
krb5kdc: starting...
W20260806 10:52:56.732746 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:52:56 dist-test-slave-0b58 krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:52:57.033471 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44481
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:44481,127.16.21.61:35053,127.16.21.60:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:52:57.353878 6687 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:52:57.354154 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:57.354239 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:57.354317 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:57.361021 6687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:57.361094 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:57.361145 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:57.361191 6687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:57.361266 6687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:57.370371 6687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:44481,127.16.21.61:35053,127.16.21.60:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:44481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:57.371371 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:57.372531 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:57.383417 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:57.383488 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:57.383529 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:57.384507 6687 server_base.cc:1034] running on GCE node
I20260806 10:52:57.385134 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:57.386315 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:57.387559 6687 hybrid_clock.cc:648] HybridClock initialized: now 1786013577387491 us; error 33 us; skew 500 ppm
I20260806 10:52:57.387849 6687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:57 dist-test-slave-0b58 krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:57.390632 6687 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:52:57.392063 6687 webserver.cc:460] Webserver started at http://127.16.21.62:46845/ using document root <none> and password file <none>
I20260806 10:52:57.392920 6687 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:57.393033 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:57.393419 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:57.397405 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "4981084d3f414b25871cf97d1ca455ce"
format_stamp: "Formatted at 2026-08-06 10:52:57 on dist-test-slave-0b58"
server_key: "d5f80a757fdee789439b390e2d0316f2"
server_key_iv: "222734251748f755c1c1e10498eb812d"
server_key_version: "encryptionkey@0"
I20260806 10:52:57.398226 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "4981084d3f414b25871cf97d1ca455ce"
format_stamp: "Formatted at 2026-08-06 10:52:57 on dist-test-slave-0b58"
server_key: "d5f80a757fdee789439b390e2d0316f2"
server_key_iv: "222734251748f755c1c1e10498eb812d"
server_key_version: "encryptionkey@0"
I20260806 10:52:57.403590 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:52:57.412755 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:57.414173 6687 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.006s
I20260806 10:52:57.414341 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "4981084d3f414b25871cf97d1ca455ce"
format_stamp: "Formatted at 2026-08-06 10:52:57 on dist-test-slave-0b58"
server_key: "d5f80a757fdee789439b390e2d0316f2"
server_key_iv: "222734251748f755c1c1e10498eb812d"
server_key_version: "encryptionkey@0"
I20260806 10:52:57.414584 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:57.435442 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:57.440029 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:57.440321 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:57.460069 6687 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:44481
I20260806 10:52:57.460108 6754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:44481 every 8 connection(s)
I20260806 10:52:57.461251 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:52:57.465750 6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:57.467844 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6687
I20260806 10:52:57.468065 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:52:57.468763 16468 external_mini_cluster.cc:1428] Setting key ffd2205f55f4cda369b1132407293cd8
I20260806 10:52:57.475665 6755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:52:57 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:57.500425 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.479541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44481 (local address 127.0.0.1:36604)
0806 10:52:57.480502 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.480555 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:57.483452 (+ 2897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:57.485587 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:57.485595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:57.486420 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:57.486864 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:57.486882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.488567 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.488576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:57.489586 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:57.489598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.490471 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.490478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:57.490703 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:57.491315 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:57.491335 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:57.493344 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:57.496564 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:57.496576 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:57.496586 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:57.496945 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:57.497337 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:57.497341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:57.497345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:57.497499 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:57.498197 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:57.498204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:57.498394 (+ 190us) client_negotiation.cc:769] Sending connection context
0806 10:52:57.498630 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 10:52:57.499194 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":138,"thread_start_us":410,"threads_started":1}
I20260806 10:52:57.500406 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.480883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.16.21.62:44481)
0806 10:52:57.484785 (+ 3902us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:57.484795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:57.484830 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:52:57.484990 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:57.484999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:57.485303 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:57.485515 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:57.487074 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.488408 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.489730 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.490248 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.490436 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:57.493579 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:57.493587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:57.493597 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:57.493622 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:57.496451 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:57.497068 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:57.497076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:57.497079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:57.497214 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:57.497600 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:57.497607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:57.497610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:57.498048 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:57.498255 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:57.498740 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 10:52:57.499081 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":3706,"thread_start_us":3570,"threads_started":1}
I20260806 10:52:57.503197 6755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:52:57.504302 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.503900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46546)
0806 10:52:57.504050 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.504157 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 10:52:57.504612 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.21.61@KRBTEST.COM; defaulting to no policy
I20260806 10:52:57.506120 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.505781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46548)
0806 10:52:57.505935 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.506004 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:52:57.506374 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35053: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab.
Entry for principal kudu/127.16.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab.
I20260806 10:52:57.542659 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:35053
--webserver_interface=127.16.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab
--principal=kudu/127.16.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:44481,127.16.21.61:35053,127.16.21.60:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:52:57.555662 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } attempt: 1
I20260806 10:52:57.556730 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.556333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46564)
0806 10:52:57.556514 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.556602 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:52:57.557637 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.557289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46578)
0806 10:52:57.557422 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.557493 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 10:52:57.557813 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35053: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
I20260806 10:52:57.626273 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } attempt: 2
I20260806 10:52:57.627305 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.626884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46584)
0806 10:52:57.627073 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.627161 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 10:52:57.628306 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.627914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46600)
0806 10:52:57.628047 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.628133 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 10:52:57.628471 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35053: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
I20260806 10:52:57.719683 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } attempt: 3
I20260806 10:52:57.720815 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.720380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46610)
0806 10:52:57.720566 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.720673 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 10:52:57.721763 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.721422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46620)
0806 10:52:57.721561 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.721630 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:52:57.721935 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35053: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
W20260806 10:52:57.857081 6762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:52:57.857403 6762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:57.857520 6762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:57.857664 6762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:57.864158 6762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:57.864295 6762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:57.864398 6762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:57.864486 6762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:57.864574 6762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:57.865145 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } attempt: 4
I20260806 10:52:57.866350 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.865766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46622)
0806 10:52:57.866060 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.866181 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260806 10:52:57.867990 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.867547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46624)
0806 10:52:57.867695 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.867815 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:52:57.868213 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:35053: Network error: Client connection negotiation failed: client connection to 127.16.21.61:35053: connect: Connection refused (error 111)
I20260806 10:52:57.876524 6762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:44481,127.16.21.61:35053,127.16.21.60:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab
--principal=kudu/127.16.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:52:57.877497 6762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:57.878644 6762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:57.889245 6767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:57.889302 6768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:57.889868 6770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:57.890442 6762 server_base.cc:1034] running on GCE node
I20260806 10:52:57.891099 6762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:57.892194 6762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:57.893390 6762 hybrid_clock.cc:648] HybridClock initialized: now 1786013577893324 us; error 41 us; skew 500 ppm
I20260806 10:52:57.893733 6762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:57 dist-test-slave-0b58 krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:57.896665 6762 init.cc:377] Logged in from keytab as kudu/127.16.21.61@KRBTEST.COM (short username kudu)
I20260806 10:52:57.897995 6762 webserver.cc:460] Webserver started at http://127.16.21.61:41597/ using document root <none> and password file <none>
I20260806 10:52:57.898578 6762 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:57.898660 6762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:57.898902 6762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:57.901808 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data/instance:
uuid: "c328cb9320a04cf9970bf7d67f981aa8"
format_stamp: "Formatted at 2026-08-06 10:52:57 on dist-test-slave-0b58"
server_key: "5b4701f58e21658b2e39b0385245fe0a"
server_key_iv: "af07aaa2d0c82e84523e2798a0cbf710"
server_key_version: "encryptionkey@0"
I20260806 10:52:57.902493 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal/instance:
uuid: "c328cb9320a04cf9970bf7d67f981aa8"
format_stamp: "Formatted at 2026-08-06 10:52:57 on dist-test-slave-0b58"
server_key: "5b4701f58e21658b2e39b0385245fe0a"
server_key_iv: "af07aaa2d0c82e84523e2798a0cbf710"
server_key_version: "encryptionkey@0"
I20260806 10:52:57.907104 6762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 10:52:57.910305 6777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:57.911468 6762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 10:52:57.911625 6762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal
uuid: "c328cb9320a04cf9970bf7d67f981aa8"
format_stamp: "Formatted at 2026-08-06 10:52:57 on dist-test-slave-0b58"
server_key: "5b4701f58e21658b2e39b0385245fe0a"
server_key_iv: "af07aaa2d0c82e84523e2798a0cbf710"
server_key_version: "encryptionkey@0"
I20260806 10:52:57.911810 6762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:57.934897 6762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:57.938310 6762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:57.938561 6762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:57.955722 6762 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.61:35053
I20260806 10:52:57.955762 6829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.61:35053 every 8 connection(s)
I20260806 10:52:57.956924 6762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
I20260806 10:52:57.958793 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6762
I20260806 10:52:57.958971 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-1/wal/instance
I20260806 10:52:57.959478 16468 external_mini_cluster.cc:1428] Setting key 716d2bdfa40b4fa104139a12786fd420
I20260806 10:52:57.961135 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:57.970481 6830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:52:57 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:57.991390 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.973664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.16.21.62:44481)
0806 10:52:57.973891 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:57.973900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:57.974987 (+ 1087us) server_negotiation.cc:407] Connection header received
0806 10:52:57.976818 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:57.976826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:57.976922 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:57.977022 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:57.978868 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.979522 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.981784 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.982155 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.982239 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:57.986127 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:57.986131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:57.986134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:57.986172 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:57.988772 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:57.989434 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:57.989438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:57.989440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:57.989545 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:57.989929 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:57.989933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:57.989940 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:57.990188 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:57.990331 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:57.990901 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 10:52:57.991103 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:52:57.992117 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.974016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44481 (local address 127.0.0.1:36616)
0806 10:52:57.974822 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.974868 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:57.976573 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:57.977275 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:57.977286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:57.977977 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:57.978698 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:57.978724 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.979715 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.979722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:57.981648 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:57.981665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.982354 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.982363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:57.982694 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:57.983346 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:57.983376 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:57.985932 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:57.988902 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:57.988910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:57.988922 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:57.989316 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:57.989643 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:57.989648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:57.989651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:57.989820 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:57.990307 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:57.990313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:57.990585 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 10:52:57.990811 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 10:52:57.991201 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":343,"threads_started":1}
I20260806 10:52:57.993708 6830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.60@KRBTEST.COM" created.
Aug 06 10:52:57 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:52:58.007344 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.994441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46626)
0806 10:52:57.994637 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:57.994655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:57.994741 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:57.995565 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:57.995586 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:57.995985 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:57.996103 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:57.996111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.997642 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.997647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:57.998614 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:57.998626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.999367 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.999372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:57.999603 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.000103 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.000118 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.001975 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.004943 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.004949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.004953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.005281 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.005675 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.005697 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.005701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.005826 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.006396 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.006401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.006549 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.006681 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.006949 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":35}
I20260806 10:52:58.007463 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:57.994640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.16.21.61:35053)
0806 10:52:57.995212 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:57.995218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:57.995240 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:52:57.995286 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:57.995290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:57.995418 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:57.995519 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:57.996291 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.997485 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.998749 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:57.999228 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:57.999361 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.002167 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.002177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.002186 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.002211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.004824 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.005399 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.005410 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.005412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.005524 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.005928 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.005935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.005938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.006252 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.006472 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.006820 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.007056 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":451,"thread_start_us":343,"threads_started":1}
I20260806 10:52:58.009524 6830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:52:58.010653 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.010236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39240)
0806 10:52:58.010400 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.010515 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 10:52:58.010903 6779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.012393 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.012045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39254)
0806 10:52:58.012170 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.012247 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 10:52:58.012619 6830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab.
Entry for principal kudu/127.16.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab.
I20260806 10:52:58.034905 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:33205
--webserver_interface=127.16.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab
--principal=kudu/127.16.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:44481,127.16.21.61:35053,127.16.21.60:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:52:58.061945 6830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 1
I20260806 10:52:58.062997 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.062522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39258)
0806 10:52:58.062719 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.062833 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 10:52:58.063994 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.063624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39260)
0806 10:52:58.063778 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.063867 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 10:52:58.064193 6830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.132457 6830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 2
I20260806 10:52:58.133824 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.133177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39262)
0806 10:52:58.133436 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.133588 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 10:52:58.134871 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.134490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39266)
0806 10:52:58.134651 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.134741 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:52:58.135121 6830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.167497 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } attempt: 5
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:52:58.182247 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.168476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.16.21.61:35053)
0806 10:52:58.168661 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.168665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.168683 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:52:58.168755 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.168759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.168821 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.168903 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.169957 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.170644 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.172391 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.172848 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.172937 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.177009 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.177014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.177018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.177074 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.179736 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.180513 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.180516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.180519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.180631 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.181053 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.181056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.181058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.181329 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.181433 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.181885 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.182031 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:52:58.182505 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.168149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46638)
0806 10:52:58.168394 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.168416 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:58.168550 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:58.168978 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:58.168983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:58.169490 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:58.169783 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.169801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.170794 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.170802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.172230 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.172246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.172992 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.173000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.173303 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.173977 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.174000 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.176795 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.179878 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.179884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.179888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.180405 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.180741 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.180746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.180749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.180929 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.181450 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.181455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.181631 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.181788 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.182052 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 10:52:58.183601 6755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:52:58.184731 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.184369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39276)
0806 10:52:58.184510 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.184580 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:52:58.185705 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.185420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39290)
0806 10:52:58.185541 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.185615 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:52:58.185910 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.226342 6830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 3
I20260806 10:52:58.227746 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.227059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39302)
0806 10:52:58.227393 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.227541 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260806 10:52:58.229214 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.228634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39304)
0806 10:52:58.228954 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.229070 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260806 10:52:58.229517 6830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.237103 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 1
I20260806 10:52:58.238020 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.237641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39310)
0806 10:52:58.237815 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.237887 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 10:52:58.238974 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.238657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39324)
0806 10:52:58.238784 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.238848 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:52:58.239240 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.307505 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 2
I20260806 10:52:58.308650 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.308192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39328)
0806 10:52:58.308406 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.308504 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 10:52:58.309589 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.309297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39334)
0806 10:52:58.309437 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.309497 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:52:58.309813 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
W20260806 10:52:58.358587 6837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:52:58.358857 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:52:58.358916 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:52:58.358984 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:52:58.365407 6837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:52:58.365476 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:52:58.365528 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:52:58.365574 6837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:52:58.365629 6837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:52:58.372722 6830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 4
I20260806 10:52:58.373747 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.373268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39346)
0806 10:52:58.373518 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.373598 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 10:52:58.374722 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.374387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39352)
0806 10:52:58.374536 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.374599 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 10:52:58.374776 6837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:44481,127.16.21.61:35053,127.16.21.60:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab
--principal=kudu/127.16.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
W20260806 10:52:58.374910 6830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.375833 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:52:58.376953 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:52:58.387629 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:58.387660 6845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:52:58.387599 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:52:58.388590 6837 server_base.cc:1034] running on GCE node
I20260806 10:52:58.389286 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:52:58.390496 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:52:58.391726 6837 hybrid_clock.cc:648] HybridClock initialized: now 1786013578391661 us; error 35 us; skew 500 ppm
I20260806 10:52:58.392010 6837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:52:58.395004 6837 init.cc:377] Logged in from keytab as kudu/127.16.21.60@KRBTEST.COM (short username kudu)
I20260806 10:52:58.396360 6837 webserver.cc:460] Webserver started at http://127.16.21.60:38363/ using document root <none> and password file <none>
I20260806 10:52:58.397013 6837 fs_manager.cc:359] Metadata directory not provided
I20260806 10:52:58.397138 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:52:58.397444 6837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:52:58.400228 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data/instance:
uuid: "d096f280c6254be18000fd8a4be6b178"
format_stamp: "Formatted at 2026-08-06 10:52:58 on dist-test-slave-0b58"
server_key: "85dcb6277ab67a2dc011b6abd3cac0e0"
server_key_iv: "1c3d658638446ee35b1f78a014259807"
server_key_version: "encryptionkey@0"
I20260806 10:52:58.400993 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal/instance:
uuid: "d096f280c6254be18000fd8a4be6b178"
format_stamp: "Formatted at 2026-08-06 10:52:58 on dist-test-slave-0b58"
server_key: "85dcb6277ab67a2dc011b6abd3cac0e0"
server_key_iv: "1c3d658638446ee35b1f78a014259807"
server_key_version: "encryptionkey@0"
I20260806 10:52:58.405555 6837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:52:58.408885 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:52:58.410132 6837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:52:58.410310 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal
uuid: "d096f280c6254be18000fd8a4be6b178"
format_stamp: "Formatted at 2026-08-06 10:52:58 on dist-test-slave-0b58"
server_key: "85dcb6277ab67a2dc011b6abd3cac0e0"
server_key_iv: "1c3d658638446ee35b1f78a014259807"
server_key_version: "encryptionkey@0"
I20260806 10:52:58.410498 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:52:58.416049 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 3
I20260806 10:52:58.416987 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.416602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39366)
0806 10:52:58.416780 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.416854 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:52:58.417930 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.417558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39382)
0806 10:52:58.417757 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.417826 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 10:52:58.418120 6755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:33205: Network error: Client connection negotiation failed: client connection to 127.16.21.60:33205: connect: Connection refused (error 111)
I20260806 10:52:58.442420 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:52:58.446439 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:52:58.446691 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:52:58.464474 6837 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.60:33205
I20260806 10:52:58.464498 6906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.60:33205 every 8 connection(s)
I20260806 10:52:58.465637 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
I20260806 10:52:58.469111 6907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:52:58.471505 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 6837
I20260806 10:52:58.471673 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013221924036-16468-0/minicluster-data/master-2/wal/instance
I20260806 10:52:58.472057 16468 external_mini_cluster.cc:1428] Setting key aff69c0d509c5007ea3b9c81f9e0eaca
I20260806 10:52:58.475833 6907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:58.492794 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.478725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.16.21.62:44481)
0806 10:52:58.478936 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.478944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.479710 (+ 766us) server_negotiation.cc:407] Connection header received
0806 10:52:58.479882 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.479887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.479958 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.480053 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.481316 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.482011 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.483616 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.484043 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.484133 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.487684 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.487688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.487691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.487725 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.490137 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.490789 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.490795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.490798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.490968 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.491383 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.491386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.491388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.491621 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.491775 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.492388 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.492582 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:52:58.493757 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.478754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:44481 (local address 127.0.0.1:36618)
0806 10:52:58.479564 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.479603 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:58.479751 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:58.480102 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:58.480106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:58.480875 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:58.481173 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.481186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.482154 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.482157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.483460 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.483493 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.484309 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.484320 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.484534 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.485177 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.485202 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.487511 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.490235 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.490242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.490248 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.490648 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.491086 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.491090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.491094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.491263 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.491752 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.491763 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.492004 (+ 241us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.492232 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.492642 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":121,"thread_start_us":362,"threads_started":1}
I20260806 10:52:58.493803 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.473842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.16.21.60:33205)
0806 10:52:58.474588 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.474597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.474634 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:52:58.476128 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.476146 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.476585 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.476877 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.478653 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.480344 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.481839 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.482554 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.482662 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.487133 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.487140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.487152 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.487179 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.489902 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.490532 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.490539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.490542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.490726 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.491086 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.491090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.491094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.491612 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.491904 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.492175 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.492533 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"spinlock_wait_cycles":16128,"thread_start_us":336,"threads_started":1}
I20260806 10:52:58.495348 6907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:52:58.509697 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.496089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.16.21.61:35053)
0806 10:52:58.496319 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.496325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.496477 (+ 152us) server_negotiation.cc:407] Connection header received
0806 10:52:58.496613 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.496617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.496697 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.496800 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.497604 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.498627 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.500062 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.500719 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.500845 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.504146 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.504150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.504153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.504184 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.507086 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.507894 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.507898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.507900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.508059 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.508517 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.508521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.508523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.508801 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.508907 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.509304 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.509447 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:52:58.509841 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.496103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:35053 (local address 127.0.0.1:46646)
0806 10:52:58.496367 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.496382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:58.496498 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:58.496835 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:58.496838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:58.497319 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:58.497451 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.497459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.498741 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.498749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.499936 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.499947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.500844 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.500848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.500996 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.501587 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.501611 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.503967 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.507240 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.507247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.507254 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.507765 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.508190 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.508197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.508202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.508407 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.508917 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.508921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.509082 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.509236 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.509495 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260806 10:52:58.510864 6907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:52:58.512794 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.496589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.16.21.62:44481)
0806 10:52:58.496746 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.496749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.496876 (+ 127us) server_negotiation.cc:407] Connection header received
0806 10:52:58.497019 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.497026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.497100 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.497199 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.498512 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.499568 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.501112 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.501690 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.501833 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.506519 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.506523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.506526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.506554 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.509802 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.510427 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.510434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.510437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.510571 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.510923 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.510928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.510932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.511248 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.511527 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.512398 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.512555 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:52:58.523869 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.511594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39402)
0806 10:52:58.511740 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.511756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:58.511839 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:58.512123 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:58.512126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:58.512522 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:58.512645 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.512653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.513836 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.513843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.515080 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.515091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.515830 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.515836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.516059 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.516605 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.516620 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.518490 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.521655 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.521659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.521661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.522066 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.522465 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.522468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.522470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.522584 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.523051 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.523056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.523212 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.523362 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.523624 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 10:52:58.524178 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.511517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.16.21.60:33205)
0806 10:52:58.511667 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.511674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.511819 (+ 145us) server_negotiation.cc:407] Connection header received
0806 10:52:58.511906 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.511910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.511980 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.512091 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.512775 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.513723 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.515230 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.515699 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.515822 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.518678 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.518681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.518684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.518725 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.521529 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.522184 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.522189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.522193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.522355 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.522694 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.522701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.522705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.522942 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.523084 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.523484 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.523812 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":53}
I20260806 10:52:58.528236 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.514288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.16.21.61:35053)
0806 10:52:58.514447 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.514454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.514471 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:52:58.514583 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.514588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.514658 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.514779 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.515861 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.516679 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.517849 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.518322 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.518447 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.522705 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.522711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.522715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.522750 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.525844 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.526584 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.526587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.526589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.526711 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.527083 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.527086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.527088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.527325 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.527516 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.527832 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.527982 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:52:58.529727 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: Bootstrap starting.
I20260806 10:52:58.532671 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:52:58.533226 6907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:58.534246 6907 log.cc:824] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:58.537155 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: No bootstrap required, opened a new log
I20260806 10:52:58.535324 6730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4981084d3f414b25871cf97d1ca455ce"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36620
I20260806 10:52:58.547676 6907 raft_consensus.cc:348] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.548051 6907 raft_consensus.cc:729] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d096f280c6254be18000fd8a4be6b178, State: Initialized, Role: FOLLOWER
I20260806 10:52:58.548540 6907 consensus_queue.cc:260] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.549408 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.549714 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.550498 6907 sys_catalog.cc:564] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:52:58.558921 6924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:52:58.559013 6924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:52:58.557212 6837 master_runner.cc:186] Error getting master registration for 127.16.21.62:44481: OK, instance_id { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" instance_seqno: 1786013577454473 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:52:58.568136 6837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:52:58.595311 6755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 4
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
I20260806 10:52:58.610687 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.595984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39416)
0806 10:52:58.596167 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.596187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:58.596327 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:58.596753 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:58.596763 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:58.597210 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:58.597405 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.597435 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.598872 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.598877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.600373 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.600388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.601266 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.601272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.601463 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.602093 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.602126 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.604572 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.608305 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.608310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.608314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.608681 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.609047 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.609052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.609056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.609201 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.609765 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.609770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.609932 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.610066 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.610340 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 10:52:58.610695 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.596065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.16.21.60:33205)
0806 10:52:58.596223 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.596230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.596281 (+ 51us) server_negotiation.cc:407] Connection header received
0806 10:52:58.596518 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.596528 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.596605 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.596711 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.597566 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.598717 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.600509 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.601137 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.601256 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.604752 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.604767 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.604772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.604808 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.608158 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.608819 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.608824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.608828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.608942 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.609313 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.609318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.609321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.609651 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.609808 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.610175 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.610361 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:52:58.615923 6755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce: Bootstrap starting.
I20260806 10:52:58.619670 6755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:58.620899 6755 log.cc:824] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:58.623684 6755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce: No bootstrap required, opened a new log
I20260806 10:52:58.628517 6755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.628813 6755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4981084d3f414b25871cf97d1ca455ce, State: Initialized, Role: FOLLOWER
I20260806 10:52:58.629313 6755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.630216 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.630560 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.631223 6755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:58.631281 6730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 10:52:58.631440 6730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:58.635455 6730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.638437 6730 leader_election.cc:288] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [CANDIDATE]: Term 1 election: Requested vote from peers c328cb9320a04cf9970bf7d67f981aa8 (127.16.21.61:35053), d096f280c6254be18000fd8a4be6b178 (127.16.21.60:33205)
I20260806 10:52:58.638314 6880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4981084d3f414b25871cf97d1ca455ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d096f280c6254be18000fd8a4be6b178"
I20260806 10:52:58.638885 6880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:58.642802 6880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4981084d3f414b25871cf97d1ca455ce in term 1.
I20260806 10:52:58.643544 6704 leader_election.cc:302] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4981084d3f414b25871cf97d1ca455ce, d096f280c6254be18000fd8a4be6b178; no voters:
I20260806 10:52:58.644093 6927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:52:58.649932 6927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [term 1 LEADER]: Becoming Leader. State: Replica: 4981084d3f414b25871cf97d1ca455ce, State: Running, Role: LEADER
I20260806 10:52:58.650506 6927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.659627 6932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4981084d3f414b25871cf97d1ca455ce. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.660027 6932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:58.661285 6936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:52:58.674089 6830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } attempt: 5
Aug 06 10:52:58 dist-test-slave-0b58 krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
W20260806 10:52:58.693346 6708 tablet.cc:2369] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:58.696359 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.674572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:33205 (local address 127.0.0.1:39424)
0806 10:52:58.674901 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 10:52:58.674921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:52:58.675020 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:52:58.675790 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:52:58.675795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:52:58.676316 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:52:58.676535 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.676548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.678032 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.678037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.679452 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:52:58.679467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.680745 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.680751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:52:58.680966 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.686053 (+ 5087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:52:58.686076 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:52:58.688816 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:52:58.692514 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.692519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.692523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.693051 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.694480 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:52:58.694486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:52:58.694490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:52:58.694660 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:52:58.695304 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:52:58.695312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:52:58.695459 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 10:52:58.695637 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 10:52:58.695942 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260806 10:52:58.697304 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.675201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.16.21.60:33205)
0806 10:52:58.675352 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.675358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.675374 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:52:58.675428 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.675433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.675514 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.675598 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.676733 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.677849 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.679620 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.680377 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.680532 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.689003 (+ 8471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.689011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.689018 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.689052 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.692386 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.694191 (+ 1805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.694196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.694199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.694349 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.694814 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.694819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.694822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.695115 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.695240 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.696779 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.696995 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 10:52:58.698689 6858 tablet.cc:2369] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:52:58.704217 6936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:52:58.704631 6830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: Bootstrap starting.
I20260806 10:52:58.708398 6830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: Neither blocks nor log segments found. Creating new log.
I20260806 10:52:58.709861 6830 log.cc:824] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:52:58.718484 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.670846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.16.21.61:35053)
0806 10:52:58.671040 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.671046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.688815 (+ 17769us) server_negotiation.cc:407] Connection header received
0806 10:52:58.689050 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.689054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.689123 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.689189 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.691594 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.692613 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.697108 (+ 4495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.697720 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.697823 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.708425 (+ 10602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.708431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.708435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.708471 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.713181 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.714931 (+ 1750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.714936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.714939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.715197 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.715886 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.715891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.715893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.716465 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.716786 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.717710 (+ 924us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.718015 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:52:58.726104 6830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: No bootstrap required, opened a new log
I20260806 10:52:58.734498 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.675938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.16.21.60:33205)
0806 10:52:58.680676 (+ 4738us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.680681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.693343 (+ 12662us) server_negotiation.cc:407] Connection header received
0806 10:52:58.693519 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.693525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.693601 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.693709 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.700367 (+ 6658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.701519 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.710589 (+ 9070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.711179 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.711285 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.722505 (+ 11220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.722514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.722521 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.722568 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.726632 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.727464 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.727469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.727472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.727642 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.728678 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.728685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.728691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.729489 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.730047 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.732958 (+ 2911us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.733453 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4617,"thread_start_us":4467,"threads_started":1}
I20260806 10:52:58.736760 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:52:58.673534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.16.21.62:44481)
0806 10:52:58.673748 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.673756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.692772 (+ 19016us) server_negotiation.cc:407] Connection header received
0806 10:52:58.693900 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.693906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.693988 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.694069 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.696625 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.697655 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.717889 (+ 20234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:52:58.718855 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:52:58.718980 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:52:58.722444 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:52:58.722452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:52:58.722457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:52:58.722495 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:52:58.726082 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.734103 (+ 8021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.734112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.734119 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.734359 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:52:58.734867 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:52:58.734875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:52:58.734881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:52:58.735286 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:52:58.735523 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 10:52:58.736071 (+ 548us) server_negotiation.cc:299] Negotiation successful
0806 10:52:58.736399 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:52:58.743240 6830 raft_consensus.cc:348] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.743638 6830 raft_consensus.cc:729] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c328cb9320a04cf9970bf7d67f981aa8, State: Initialized, Role: FOLLOWER
I20260806 10:52:58.744125 6830 consensus_queue.cc:260] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } }
I20260806 10:52:58.745108 6950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.745509 6950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.746140 6830 sys_catalog.cc:564] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:52:58.746183 6805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4981084d3f414b25871cf97d1ca455ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c328cb9320a04cf9970bf7d67f981aa8"
I20260806 10:52:58.746634 6805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:52:58.749778 6805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4981084d3f414b25871cf97d1ca455ce in term 1.
I20260806 10:52:58.757221 6880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [term 1 FOLLOWER]: Refusing update from remote peer 4981084d3f414b25871cf97d1ca455ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:58.757361 6805 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [term 1 FOLLOWER]: Refusing update from remote peer 4981084d3f414b25871cf97d1ca455ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:52:58.758100 6927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:58.758893 6927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:52:58.773274 6950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4981084d3f414b25871cf97d1ca455ce. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.773566 6950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.775038 6948 mvcc.cc:205] Tried to move back new op lower bound from 7315511618580688896 to 7315511618158985216. Current Snapshot: MvccSnapshot[applied={T|T < 7315511618580688896}]
I20260806 10:52:58.776556 6950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.776785 6950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.777052 6955 mvcc.cc:205] Tried to move back new op lower bound from 7315511618580688896 to 7315511618158985216. Current Snapshot: MvccSnapshot[applied={T|T < 7315511618580688896}]
I20260806 10:52:58.769695 6687 master_runner.cc:186] Error getting master registration for 127.16.21.61:35053: OK, instance_id { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" instance_seqno: 1786013577950856 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:52:58.781956 6936 catalog_manager.cc:1347] Generated new cluster ID: e57c825e498a4995b598cc9244359af7
I20260806 10:52:58.782419 6936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:52:58.782127 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.783946 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4981084d3f414b25871cf97d1ca455ce. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.784192 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.785935 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:58.786386 6687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:52:58.787065 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.787312 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce [sys.catalog]: This master's current role is: LEADER
I20260806 10:52:58.804591 6972 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 10:52:58.805298 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4981084d3f414b25871cf97d1ca455ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4981084d3f414b25871cf97d1ca455ce" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 44481 } } peers { permanent_uuid: "c328cb9320a04cf9970bf7d67f981aa8" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 35053 } } peers { permanent_uuid: "d096f280c6254be18000fd8a4be6b178" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 33205 } } }
I20260806 10:52:58.805521 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:52:58.838246 6936 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:52:58.840076 6936 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:52:58.851148 6936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4981084d3f414b25871cf97d1ca455ce: Generated new TSK 0
I20260806 10:52:58.852010 6936 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:52:58.953307 6758 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:58.940026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.16.21.62:44481)
0806 10:52:58.940206 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.940212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.950616 (+ 10404us) server_negotiation.cc:407] Connection header received
0806 10:52:58.951564 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.951571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.951657 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.951922 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.953059 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36638: BlockingRecv error: recv got EOF from 127.0.0.1:36638 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 10:52:58.955058 6943 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:58.939429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.16.21.60:33205)
0806 10:52:58.939616 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.939622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.952821 (+ 13199us) server_negotiation.cc:407] Connection header received
0806 10:52:58.953057 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.953063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.953159 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.953235 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.954784 (+ 1549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39442: BlockingRecv error: recv got EOF from 127.0.0.1:39442 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 10:52:58.957439 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:58.939840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.16.21.61:35053)
0806 10:52:58.940005 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.940010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.954644 (+ 14634us) server_negotiation.cc:407] Connection header received
0806 10:52:58.956307 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.956313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.956381 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.956459 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.957257 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46678: BlockingRecv error: recv got EOF from 127.0.0.1:46678 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 10:52:58.995301 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:58.989399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.16.21.61:35053)
0806 10:52:58.989600 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.989608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.993840 (+ 4232us) server_negotiation.cc:407] Connection header received
0806 10:52:58.994067 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:58.994072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:58.994179 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:58.994289 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:58.995021 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46694: BlockingRecv error: recv got EOF from 127.0.0.1:46694 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 10:52:59.001140 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:58.999264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.16.21.61:35053)
0806 10:52:58.999443 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:58.999450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:58.999915 (+ 465us) server_negotiation.cc:407] Connection header received
0806 10:52:59.000114 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.000118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.000189 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.000323 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.000963 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46706: BlockingRecv error: recv got EOF from 127.0.0.1:46706 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 10:52:59.007921 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.005992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.16.21.61:35053)
0806 10:52:59.006206 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.006213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.006712 (+ 499us) server_negotiation.cc:407] Connection header received
0806 10:52:59.006897 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.006902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.006980 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.007078 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.007709 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46716: BlockingRecv error: recv got EOF from 127.0.0.1:46716 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 10:52:59.016947 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.014840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.16.21.61:35053)
0806 10:52:59.015017 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.015024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.015554 (+ 530us) server_negotiation.cc:407] Connection header received
0806 10:52:59.015822 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.015828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.015897 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.015994 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.016749 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46718: BlockingRecv error: recv got EOF from 127.0.0.1:46718 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 10:52:59.029727 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.027752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.16.21.61:35053)
0806 10:52:59.027921 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.027929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.028496 (+ 567us) server_negotiation.cc:407] Connection header received
0806 10:52:59.028757 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.028762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.028827 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.028926 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.029540 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46720: BlockingRecv error: recv got EOF from 127.0.0.1:46720 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 10:52:59.050647 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.048783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.16.21.61:35053)
0806 10:52:59.048975 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.048983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.049446 (+ 463us) server_negotiation.cc:407] Connection header received
0806 10:52:59.049626 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.049631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.049704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.049803 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.050442 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46730: BlockingRecv error: recv got EOF from 127.0.0.1:46730 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 10:52:59.088141 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.086417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.16.21.61:35053)
0806 10:52:59.086573 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.086580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.086834 (+ 254us) server_negotiation.cc:407] Connection header received
0806 10:52:59.087048 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.087051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.087123 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.087229 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.087905 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46746: BlockingRecv error: recv got EOF from 127.0.0.1:46746 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260806 10:52:59.157645 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.155679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.16.21.61:35053)
0806 10:52:59.155920 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.155927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.156237 (+ 310us) server_negotiation.cc:407] Connection header received
0806 10:52:59.156435 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.156442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.156547 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.156676 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.157418 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46754: BlockingRecv error: recv got EOF from 127.0.0.1:46754 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260806 10:52:59.291042 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.289439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.16.21.61:35053)
0806 10:52:59.289604 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.289608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.290000 (+ 392us) server_negotiation.cc:407] Connection header received
0806 10:52:59.290138 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.290141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.290203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.290283 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.290873 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46766: BlockingRecv error: recv got EOF from 127.0.0.1:46766 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 10:52:59.553435 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:52:59.551422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.16.21.61:35053)
0806 10:52:59.551655 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 10:52:59.551662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:52:59.551921 (+ 259us) server_negotiation.cc:407] Connection header received
0806 10:52:59.552132 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:52:59.552146 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:52:59.552233 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:52:59.552370 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:52:59.553199 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46778: BlockingRecv error: recv got EOF from 127.0.0.1:46778 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:52:59.560802 6924 catalog_manager.cc:1259] Loaded cluster ID: e57c825e498a4995b598cc9244359af7
I20260806 10:52:59.560883 6924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: loading cluster ID for follower catalog manager: success
I20260806 10:52:59.563103 6924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: acquiring CA information for follower catalog manager: success
I20260806 10:52:59.564826 6924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d096f280c6254be18000fd8a4be6b178: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:52:59.809091 6972 catalog_manager.cc:1259] Loaded cluster ID: e57c825e498a4995b598cc9244359af7
I20260806 10:52:59.809223 6972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: loading cluster ID for follower catalog manager: success
I20260806 10:52:59.811945 6972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: acquiring CA information for follower catalog manager: success
I20260806 10:52:59.813868 6972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c328cb9320a04cf9970bf7d67f981aa8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:53:00.088554 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:00.081975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.16.21.61:35053)
0806 10:53:00.082786 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:00.082791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:00.082806 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:53:00.082872 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:00.082876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:00.083018 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:00.083165 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:00.083669 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.084699 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.086032 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.086511 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.086644 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:00.087168 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:00.087751 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:00.087909 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:00.087995 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 10:53:00.088079 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":500,"threads_started":1}
I20260806 10:53:00.101378 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:00.094769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.16.21.60:33205)
0806 10:53:00.095325 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:00.095331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:00.095697 (+ 366us) server_negotiation.cc:407] Connection header received
0806 10:53:00.095925 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:00.095929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:00.096087 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:00.096217 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:00.096677 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.097362 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.099286 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.099657 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.099742 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:00.100328 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:00.100793 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:00.100940 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:00.101028 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 10:53:00.101090 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
I20260806 10:53:00.119596 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:00.107830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.16.21.60:33205)
0806 10:53:00.112776 (+ 4946us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:00.112782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:00.112801 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:53:00.112858 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:00.112864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:00.112943 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:00.113023 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:00.113662 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.114833 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.117280 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.117892 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.117984 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:00.118738 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:00.118866 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:00.118927 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:00.119149 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 10:53:00.119237 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4824}
I20260806 10:53:00.126396 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:00.107404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.16.21.61:35053)
0806 10:53:00.107549 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:00.107554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:00.120440 (+ 12886us) server_negotiation.cc:407] Connection header received
0806 10:53:00.120679 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:00.120685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:00.120760 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:00.120816 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:00.121329 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.122415 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.124585 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.125062 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.125193 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:00.125633 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:00.125783 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:00.125882 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:00.126018 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 10:53:00.126091 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:53:00.127009 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:00.107671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.16.21.62:44481)
0806 10:53:00.120819 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:00.120825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:00.120843 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:53:00.120953 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:00.120958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:00.121155 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:00.122029 (+ 874us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:00.122166 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.122923 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.124615 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:00.125300 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:00.125433 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:00.125915 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:00.126378 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:00.126491 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:00.126617 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 10:53:00.126699 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13025,"thread_start_us":13645,"threads_started":1}
I20260806 10:53:00.132799 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6687
I20260806 10:53:00.133765 6750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:00.255113 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6687
I20260806 10:53:00.279556 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6762
I20260806 10:53:00.280793 6825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:00.381707 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6762
I20260806 10:53:00.405153 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 6837
I20260806 10:53:00.406242 6901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:00.512393 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 6837
2026-08-06T10:53:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4887 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:00 dist-test-slave-0b58 krb5kdc[7063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:00 dist-test-slave-0b58 krb5kdc[7063](info): set up 2 sockets
Aug 06 10:53:00 dist-test-slave-0b58 krb5kdc[7063](info): commencing operation
krb5kdc: starting...
W20260806 10:53:01.616055 16468 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/test-user.keytab.
Aug 06 10:53:01 dist-test-slave-0b58 krb5kdc[7063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
Entry for principal kudu/127.16.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab.
I20260806 10:53:01.923779 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38501
--webserver_interface=127.16.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:38501,127.16.21.61:39947,127.16.21.60:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:02.254577 7076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:53:02.254951 7076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:02.255091 7076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:02.255234 7076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:02.262667 7076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:02.262794 7076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:02.262910 7076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:02.262996 7076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:02.263090 7076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:02.272590 7076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:38501,127.16.21.61:39947,127.16.21.60:45491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.62.keytab
--principal=kudu/127.16.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.62:38501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:02.273789 7076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:02.275077 7076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:02.286559 7084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:02.286638 7081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:02.286605 7082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:02.287629 7076 server_base.cc:1034] running on GCE node
I20260806 10:53:02.288335 7076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:02.289583 7076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:02.290815 7076 hybrid_clock.cc:648] HybridClock initialized: now 1786013582290761 us; error 41 us; skew 500 ppm
I20260806 10:53:02.291114 7076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:02 dist-test-slave-0b58 krb5kdc[7063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:02.294054 7076 init.cc:377] Logged in from keytab as kudu/127.16.21.62@KRBTEST.COM (short username kudu)
I20260806 10:53:02.295440 7076 webserver.cc:460] Webserver started at http://127.16.21.62:39259/ using document root <none> and password file <none>
I20260806 10:53:02.296084 7076 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:02.296200 7076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:02.296530 7076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:02.299116 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data/instance:
uuid: "ee9eb519c85446ae8ec75a2c0ba84ade"
format_stamp: "Formatted at 2026-08-06 10:53:02 on dist-test-slave-0b58"
server_key: "fa89c119facc048cf83d6c4c622e231d"
server_key_iv: "130670afa7e01ab4e465451f6e254755"
server_key_version: "encryptionkey@0"
I20260806 10:53:02.299913 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal/instance:
uuid: "ee9eb519c85446ae8ec75a2c0ba84ade"
format_stamp: "Formatted at 2026-08-06 10:53:02 on dist-test-slave-0b58"
server_key: "fa89c119facc048cf83d6c4c622e231d"
server_key_iv: "130670afa7e01ab4e465451f6e254755"
server_key_version: "encryptionkey@0"
I20260806 10:53:02.304663 7076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260806 10:53:02.307921 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:02.309089 7076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:53:02.309291 7076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal
uuid: "ee9eb519c85446ae8ec75a2c0ba84ade"
format_stamp: "Formatted at 2026-08-06 10:53:02 on dist-test-slave-0b58"
server_key: "fa89c119facc048cf83d6c4c622e231d"
server_key_iv: "130670afa7e01ab4e465451f6e254755"
server_key_version: "encryptionkey@0"
I20260806 10:53:02.309513 7076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:02.341724 7076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:02.345199 7076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:02.345458 7076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:02.363287 7076 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.62:38501
I20260806 10:53:02.363306 7143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.62:38501 every 8 connection(s)
I20260806 10:53:02.364522 7076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:02.367914 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:02.370338 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 7076
I20260806 10:53:02.370633 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-0/wal/instance
I20260806 10:53:02.371088 16468 external_mini_cluster.cc:1428] Setting key d0a3eb33d0e62ea6d217466648040937
I20260806 10:53:02.374141 7144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:53:02 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:53:02.395298 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.376864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.16.21.62:38501)
0806 10:53:02.378479 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:02.378488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:02.378533 (+ 45us) server_negotiation.cc:407] Connection header received
0806 10:53:02.380113 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:02.380122 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:02.380412 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:02.380594 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:02.381643 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.383356 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.384717 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.385600 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.385766 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:02.388967 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:02.388974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:02.388983 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:02.389008 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:02.391513 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:02.392343 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:02.392348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:02.392350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:02.392476 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:02.392924 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:02.392927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:02.392929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:02.393307 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:02.393444 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:02.394109 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 10:53:02.394406 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":539,"thread_start_us":348,"threads_started":1}
I20260806 10:53:02.395323 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.376865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38501 (local address 127.0.0.1:57296)
0806 10:53:02.378082 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.378140 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:02.379783 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:02.380619 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:02.380642 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:02.381127 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:02.381444 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:02.381464 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.383491 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.383495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:02.384585 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:02.384597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.385794 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.385799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:02.385909 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:02.386503 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:02.386523 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:02.388781 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:02.391634 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:02.391648 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:02.391659 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:02.392190 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:02.392576 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:02.392582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:02.392588 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:02.392815 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:02.393447 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:02.393457 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:02.393702 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 10:53:02.393990 (+ 288us) client_negotiation.cc:240] Negotiation successful
0806 10:53:02.394595 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":64,"thread_start_us":715,"threads_started":1}
I20260806 10:53:02.397861 7144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:53:02.398874 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.398415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54500)
0806 10:53:02.398587 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.398722 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 10:53:02.399143 7094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
I20260806 10:53:02.400681 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.400307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54510)
0806 10:53:02.400463 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.400543 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:53:02.400861 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:39947: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab.
Entry for principal kudu/127.16.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab.
I20260806 10:53:02.441617 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:39947
--webserver_interface=127.16.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab
--principal=kudu/127.16.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:38501,127.16.21.61:39947,127.16.21.60:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:53:02.450145 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } attempt: 1
I20260806 10:53:02.451212 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.450806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54526)
0806 10:53:02.450987 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.451077 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 10:53:02.452099 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.451790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54538)
0806 10:53:02.451921 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.451983 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:53:02.452284 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:39947: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
I20260806 10:53:02.520553 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } attempt: 2
I20260806 10:53:02.521719 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.521245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54540)
0806 10:53:02.521497 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.521580 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 10:53:02.522648 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.522311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54548)
0806 10:53:02.522444 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.522522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:53:02.522825 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:39947: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
I20260806 10:53:02.614071 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } attempt: 3
I20260806 10:53:02.615135 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.614687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54560)
0806 10:53:02.614896 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.614986 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 10:53:02.616098 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.615764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54568)
0806 10:53:02.615912 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.615978 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 10:53:02.616309 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:39947: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
I20260806 10:53:02.759611 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } attempt: 4
I20260806 10:53:02.760825 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.760309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54578)
0806 10:53:02.760615 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.760696 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260806 10:53:02.761723 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.761393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54580)
0806 10:53:02.761537 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.761611 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 10:53:02.761895 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.61:39947: Network error: Client connection negotiation failed: client connection to 127.16.21.61:39947: connect: Connection refused (error 111)
W20260806 10:53:02.764204 7151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:53:02.764575 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:02.764689 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:02.764827 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:02.771344 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:02.771452 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:02.771538 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:02.771615 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:02.771694 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:02.780395 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:38501,127.16.21.61:39947,127.16.21.60:45491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.61.keytab
--principal=kudu/127.16.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.61:39947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:02.781404 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:02.782480 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:02.793444 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:02.793537 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:02.793517 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:02.794186 7151 server_base.cc:1034] running on GCE node
I20260806 10:53:02.794914 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:02.796077 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:02.797276 7151 hybrid_clock.cc:648] HybridClock initialized: now 1786013582797190 us; error 51 us; skew 500 ppm
I20260806 10:53:02.797521 7151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:02 dist-test-slave-0b58 krb5kdc[7063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:02.800292 7151 init.cc:377] Logged in from keytab as kudu/127.16.21.61@KRBTEST.COM (short username kudu)
I20260806 10:53:02.801535 7151 webserver.cc:460] Webserver started at http://127.16.21.61:42921/ using document root <none> and password file <none>
I20260806 10:53:02.802174 7151 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:02.802322 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:02.802629 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:02.805404 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data/instance:
uuid: "8841b443b7e7404582921bf73d4ce2e2"
format_stamp: "Formatted at 2026-08-06 10:53:02 on dist-test-slave-0b58"
server_key: "5ca92475d5c172bd0db07a09264b9254"
server_key_iv: "3c7fa360124d0cb5245d29782ba7f1ae"
server_key_version: "encryptionkey@0"
I20260806 10:53:02.806133 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal/instance:
uuid: "8841b443b7e7404582921bf73d4ce2e2"
format_stamp: "Formatted at 2026-08-06 10:53:02 on dist-test-slave-0b58"
server_key: "5ca92475d5c172bd0db07a09264b9254"
server_key_iv: "3c7fa360124d0cb5245d29782ba7f1ae"
server_key_version: "encryptionkey@0"
I20260806 10:53:02.811023 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 10:53:02.814304 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:02.815455 7151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:53:02.815613 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal
uuid: "8841b443b7e7404582921bf73d4ce2e2"
format_stamp: "Formatted at 2026-08-06 10:53:02 on dist-test-slave-0b58"
server_key: "5ca92475d5c172bd0db07a09264b9254"
server_key_iv: "3c7fa360124d0cb5245d29782ba7f1ae"
server_key_version: "encryptionkey@0"
I20260806 10:53:02.815810 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:02.835019 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:02.838394 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:02.838624 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:02.855604 7151 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.61:39947
I20260806 10:53:02.855679 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.61:39947 every 8 connection(s)
I20260806 10:53:02.856710 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/data/info.pb
I20260806 10:53:02.859874 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:02.861799 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 7151
I20260806 10:53:02.861996 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-1/wal/instance
I20260806 10:53:02.862406 16468 external_mini_cluster.cc:1428] Setting key 76830e5fffeb5897279a50230c61b87e
I20260806 10:53:02.865916 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:53:02 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:53:02.888208 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.868089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.16.21.62:38501)
0806 10:53:02.868293 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:02.868300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:02.869499 (+ 1199us) server_negotiation.cc:407] Connection header received
0806 10:53:02.871109 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:02.871114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:02.871198 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:02.871297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:02.872598 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.873281 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.874894 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.875528 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.875658 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:02.881895 (+ 6237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:02.881903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:02.881907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:02.881942 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:02.885206 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:02.886079 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:02.886083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:02.886085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:02.886219 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:02.886663 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:02.886666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:02.886668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:02.886916 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:02.887030 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:02.887844 (+ 814us) server_negotiation.cc:299] Negotiation successful
0806 10:53:02.887997 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 10:53:02.889618 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.868609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38501 (local address 127.0.0.1:57302)
0806 10:53:02.869362 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.869394 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:02.870871 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:02.871513 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:02.871519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:02.871936 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:02.872445 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:02.872462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.873430 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.873440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:02.874694 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:02.874706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.876019 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.876028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:02.876373 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:02.878486 (+ 2113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:02.878519 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:02.881674 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:02.885370 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:02.885384 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:02.885402 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:02.885970 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:02.886320 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:02.886327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:02.886333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:02.886558 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:02.887050 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:02.887061 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:02.887495 (+ 434us) client_negotiation.cc:769] Sending connection context
0806 10:53:02.887764 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 10:53:02.888297 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":350,"threads_started":1}
I20260806 10:53:02.891201 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:53:02 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.21.60@KRBTEST.COM" created.
I20260806 10:53:02.904937 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.892206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.16.21.61:39947)
0806 10:53:02.892787 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:02.892792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:02.892813 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:53:02.892848 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:02.892852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:02.892985 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:02.893114 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:02.893772 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.894860 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.895935 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.896478 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.896606 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:02.899757 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:02.899768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:02.899778 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:02.899807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:02.902234 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:02.902805 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:02.902811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:02.902813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:02.902921 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:02.903320 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:02.903323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:02.903325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:02.903700 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:02.903872 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:02.904367 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 10:53:02.904640 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":456,"thread_start_us":341,"threads_started":1}
I20260806 10:53:02.905036 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.891955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54584)
0806 10:53:02.892136 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.892155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:02.892225 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:02.893110 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:02.893114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:02.893499 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:02.893615 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:02.893622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.894988 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.894993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:02.895809 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:02.895819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:02.896609 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:02.896614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:02.896794 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:02.897311 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:02.897326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:02.899572 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:02.902347 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:02.902350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:02.902352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:02.902694 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:02.903036 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:02.903039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:02.903041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:02.903197 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:02.903835 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:02.903840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:02.904001 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 10:53:02.904175 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 10:53:02.904533 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 10:53:02.907264 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:53:02.908332 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.907805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34240)
0806 10:53:02.907994 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.908128 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:53:02.908571 7171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:02.910002 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.909701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34246)
0806 10:53:02.909828 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.909904 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:53:02.910300 7220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab.
Entry for principal kudu/127.16.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab.
I20260806 10:53:02.939693 16468 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
/tmp/dist-test-taskLWW7rW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:45491
--webserver_interface=127.16.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab
--principal=kudu/127.16.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.21.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.21.62:38501,127.16.21.61:39947,127.16.21.60:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:53:02.959668 7220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 1
I20260806 10:53:02.960682 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.960250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34260)
0806 10:53:02.960438 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.960533 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 10:53:02.961578 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:02.961282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34268)
0806 10:53:02.961409 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:02.961493 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 10:53:02.961740 7220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.030037 7220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 2
I20260806 10:53:03.031188 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.030632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34282)
0806 10:53:03.030881 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.030988 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 10:53:03.032353 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.031829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34284)
0806 10:53:03.032015 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.032141 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 10:53:03.032601 7220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.061143 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } attempt: 5
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:53:03.074976 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.061744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54586)
0806 10:53:03.061917 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.061934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:03.062040 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:03.062361 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:03.062364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:03.062714 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:03.062909 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.062917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.064339 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.064344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.065215 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.065226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.066156 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.066170 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.066318 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.066920 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:03.066935 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:03.069099 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:03.072825 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.072828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.072831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.073146 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.073510 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.073513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.073515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.073632 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.074327 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:03.074330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:03.074448 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 10:53:03.074561 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 10:53:03.074758 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:53:03.075209 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.061814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.16.21.61:39947)
0806 10:53:03.062008 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.062014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.062037 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:53:03.062137 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.062143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.062220 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.062337 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.063048 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.064162 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.065370 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.066033 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.066155 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.069285 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.069292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.069296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.069334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.072701 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.073252 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.073258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.073261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.073406 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.073753 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.073760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.073765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.074217 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.074374 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.074699 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.074919 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:53:03.076133 7144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:53:03.077147 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.076796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34300)
0806 10:53:03.076944 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.077022 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 10:53:03.077986 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.077684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34302)
0806 10:53:03.077835 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.077902 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 10:53:03.078161 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.123821 7220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 3
I20260806 10:53:03.125072 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.124442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34318)
0806 10:53:03.124708 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.124838 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 10:53:03.126318 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.125764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34332)
0806 10:53:03.126009 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.126113 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260806 10:53:03.126596 7220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.129356 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 1
I20260806 10:53:03.130146 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.129787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34334)
0806 10:53:03.129939 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.130013 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:53:03.131034 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.130730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34342)
0806 10:53:03.130864 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.130927 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:53:03.131240 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.199515 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 2
I20260806 10:53:03.200605 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.200141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34344)
0806 10:53:03.200380 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.200460 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 10:53:03.201447 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.201172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34358)
0806 10:53:03.201297 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.201359 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 10:53:03.201651 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
W20260806 10:53:03.260905 7228 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:53:03.261215 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:03.261325 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:03.261463 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:03.268445 7228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:03.268517 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:03.268623 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:03.268705 7228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:03.268793 7228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:03.269810 7220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 4
I20260806 10:53:03.270951 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.270406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34368)
0806 10:53:03.270652 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.270766 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 10:53:03.271963 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.271594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34382)
0806 10:53:03.271755 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.271839 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:53:03.272217 7220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.279057 7228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.21.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.21.62:38501,127.16.21.61:39947,127.16.21.60:45491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/krb5kdc/kudu_127.16.21.60.keytab
--principal=kudu/127.16.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.21.60:45491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:03.280124 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:03.281347 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:03.292052 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:03.292117 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:03.292090 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:03.293191 7228 server_base.cc:1034] running on GCE node
I20260806 10:53:03.293931 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:03.295228 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:03.296452 7228 hybrid_clock.cc:648] HybridClock initialized: now 1786013583296398 us; error 44 us; skew 500 ppm
I20260806 10:53:03.296772 7228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:03.299551 7228 init.cc:377] Logged in from keytab as kudu/127.16.21.60@KRBTEST.COM (short username kudu)
I20260806 10:53:03.300832 7228 webserver.cc:460] Webserver started at http://127.16.21.60:44953/ using document root <none> and password file <none>
I20260806 10:53:03.301460 7228 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:03.301543 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:03.301789 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:03.304632 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data/instance:
uuid: "7036e4af9c2e420f876cacbb8eccae8b"
format_stamp: "Formatted at 2026-08-06 10:53:03 on dist-test-slave-0b58"
server_key: "08e328ca04da3687460b168b881e032d"
server_key_iv: "50d6435f59fead073a3068b52943ffee"
server_key_version: "encryptionkey@0"
I20260806 10:53:03.305452 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal/instance:
uuid: "7036e4af9c2e420f876cacbb8eccae8b"
format_stamp: "Formatted at 2026-08-06 10:53:03 on dist-test-slave-0b58"
server_key: "08e328ca04da3687460b168b881e032d"
server_key_iv: "50d6435f59fead073a3068b52943ffee"
server_key_version: "encryptionkey@0"
I20260806 10:53:03.307847 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 3
I20260806 10:53:03.308813 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.308402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34384)
0806 10:53:03.308573 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.308655 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:53:03.310007 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.309586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34392)
0806 10:53:03.309780 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.309846 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260806 10:53:03.310251 7144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.21.60:45491: Network error: Client connection negotiation failed: client connection to 127.16.21.60:45491: connect: Connection refused (error 111)
I20260806 10:53:03.310611 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 10:53:03.313777 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:03.315035 7228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:53:03.315224 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data,/tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal
uuid: "7036e4af9c2e420f876cacbb8eccae8b"
format_stamp: "Formatted at 2026-08-06 10:53:03 on dist-test-slave-0b58"
server_key: "08e328ca04da3687460b168b881e032d"
server_key_iv: "50d6435f59fead073a3068b52943ffee"
server_key_version: "encryptionkey@0"
I20260806 10:53:03.315457 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:03.334316 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:03.337882 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:03.338130 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:03.354266 7228 rpc_server.cc:307] RPC server started. Bound to: 127.16.21.60:45491
I20260806 10:53:03.354300 7294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.21.60:45491 every 8 connection(s)
I20260806 10:53:03.355427 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/data/info.pb
I20260806 10:53:03.356071 16468 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu as pid 7228
I20260806 10:53:03.356288 16468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLWW7rW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013221924036-16468-0/minicluster-data/master-2/wal/instance
I20260806 10:53:03.356806 16468 external_mini_cluster.cc:1428] Setting key 22c902e02ef01cad6c213ca1a2342907
I20260806 10:53:03.359261 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:03.366770 7295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
I20260806 10:53:03.379469 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.358361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.16.21.60:45491)
0806 10:53:03.359098 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.359108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.359143 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:53:03.360600 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.360634 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.360958 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.361226 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.363018 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.365114 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.366844 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.367825 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.367980 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.372929 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.372936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.372944 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.372975 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.375938 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.376635 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.376640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.376645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.376792 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.377120 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.377127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.377130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.377788 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.378091 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.378362 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.378698 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":366,"threads_started":1}
I20260806 10:53:03.384236 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.369121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.62:38501 (local address 127.0.0.1:57304)
0806 10:53:03.369902 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.369937 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:03.370108 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:03.370481 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:03.370486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:03.371192 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:03.371478 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.371488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.372934 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.372938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.374111 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.374123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.375188 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.375192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.375323 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.375801 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:03.375817 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:03.378204 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:03.381545 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.381557 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.381563 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.382016 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.382524 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.382528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.382531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.382687 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.383175 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:03.383181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:03.383331 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 10:53:03.383492 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 10:53:03.383785 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":418,"threads_started":1}
I20260806 10:53:03.384248 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.369192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.16.21.62:38501)
0806 10:53:03.369371 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.369379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.370013 (+ 634us) server_negotiation.cc:407] Connection header received
0806 10:53:03.370235 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.370240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.370327 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.370436 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.371627 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.372708 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.374256 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.374997 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.375153 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.378404 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.378410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.378415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.378453 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.381426 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.382168 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.382175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.382181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.382403 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.382781 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.382784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.382786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.383055 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.383197 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.383607 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.383796 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":61}
I20260806 10:53:03.385538 7295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.62@KRBTEST.COM
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
I20260806 10:53:03.396878 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.382074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.16.21.62:38501)
0806 10:53:03.382615 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.382620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.382641 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:53:03.382685 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.382688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.382768 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.382857 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.384041 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.385295 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.386721 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.387138 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.387252 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.391627 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.391632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.391636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.391669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.394283 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.395059 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.395063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.395065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.395180 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.395523 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.395526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.395528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.395745 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.395913 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.396247 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.396475 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":413,"thread_start_us":241,"threads_started":1}
I20260806 10:53:03.398437 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.386139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.61:39947 (local address 127.0.0.1:54600)
0806 10:53:03.386316 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.386333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:03.386420 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:03.386701 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:03.386708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:03.387013 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:03.387132 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.387139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.388117 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.388123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.389631 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.389647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.390291 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.390297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.390427 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.390918 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:03.390936 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:03.392894 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:03.396039 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.396042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.396044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.396502 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.396972 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.396980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.396985 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.397169 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.397665 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:03.397669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:03.397835 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 10:53:03.397936 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:53:03.398130 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 10:53:03.398716 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.386181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.16.21.61:39947)
0806 10:53:03.386382 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.386386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.386402 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:53:03.386520 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.386523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.386577 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.386648 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.387291 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.387981 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.389782 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.390145 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.390272 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.393063 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.393067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.393070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.393097 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.395926 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.396650 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.396656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.396660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.396837 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.397304 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.397308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.397311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.397546 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.397686 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.398076 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.398275 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 10:53:03.403162 7295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.21.61@KRBTEST.COM
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.60@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
I20260806 10:53:03.413540 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.398394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.16.21.61:39947)
0806 10:53:03.399212 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.399220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.399241 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:53:03.399311 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.399318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.399410 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.399523 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.400555 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.401841 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.403525 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.404228 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.404385 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.408204 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.408209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.408213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.408248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.410881 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.411628 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.411634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.411638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.411801 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.412207 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.412210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.412213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.412503 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.412805 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.413090 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.413227 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":3941,"threads_started":1}
I20260806 10:53:03.416724 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.403785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34424)
0806 10:53:03.404003 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.404018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:03.404100 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:03.404462 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:03.404466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:03.404906 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:03.405086 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.405115 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.405967 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.405973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.406805 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.406815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.408118 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.408124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.408322 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.408827 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:03.408851 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:03.410871 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:03.414359 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.414363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.414365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.414696 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.415078 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.415096 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.415100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.415237 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.415795 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:03.415800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:03.415931 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 10:53:03.416085 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 10:53:03.416383 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260806 10:53:03.417042 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.403978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.16.21.60:45491)
0806 10:53:03.404189 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.404196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.404213 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:53:03.404272 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.404278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.404343 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.404447 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.405237 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.405856 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.407428 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.407958 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.408088 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.411034 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.411041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.411045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.411080 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.414222 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.414826 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.414830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.414832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.414971 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.415350 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.415359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.415360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.415668 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.415833 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.416178 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.416560 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":83}
I20260806 10:53:03.417718 16468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:53:03.421416 7295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: Bootstrap starting.
I20260806 10:53:03.420151 7119 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320
I20260806 10:53:03.424583 7295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:03.425563 7295 log.cc:824] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:03.428028 7295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: No bootstrap required, opened a new log
I20260806 10:53:03.434638 7295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.434931 7295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7036e4af9c2e420f876cacbb8eccae8b, State: Initialized, Role: FOLLOWER
I20260806 10:53:03.435343 7295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.436147 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.436513 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.436997 7295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:53:03.445946 7314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:53:03.446058 7314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:53:03.443609 7228 master_runner.cc:186] Error getting master registration for 127.16.21.62:38501: OK, instance_id { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" instance_seqno: 1786013582358573 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:53:03.454505 7228 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:53:03.487593 7144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 4
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.62@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
I20260806 10:53:03.501175 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.488210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34438)
0806 10:53:03.488408 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.488446 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:03.488557 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:03.488873 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:03.488880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:03.489396 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:03.489602 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.489615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.490609 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.490618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.492013 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.492030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.492811 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.492817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.492999 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.493622 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:03.493647 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:03.495890 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:03.498900 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.498904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.498907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.499223 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.499562 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.499566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.499570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.499718 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.500235 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:03.500241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:03.500418 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 10:53:03.500546 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 10:53:03.500835 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 10:53:03.501191 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.488298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.16.21.60:45491)
0806 10:53:03.488484 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.488488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.488506 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:53:03.488694 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.488697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.488752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.488823 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.489757 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.490490 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.492171 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.492684 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.492775 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.496069 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.496073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.496076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.496101 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.498751 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.499327 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.499335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.499339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.499443 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.499818 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.499823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.499826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.500091 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.500300 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.500632 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.500834 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:53:03.506754 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade: Bootstrap starting.
I20260806 10:53:03.510525 7144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:03.511683 7144 log.cc:824] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:03.514801 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade: No bootstrap required, opened a new log
I20260806 10:53:03.521723 7144 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.522084 7144 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9eb519c85446ae8ec75a2c0ba84ade, State: Initialized, Role: FOLLOWER
I20260806 10:53:03.522514 7144 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.523363 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.523720 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.524503 7144 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:03.524567 7119 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 10:53:03.524770 7119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:03.527223 7119 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.528534 7119 leader_election.cc:288] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [CANDIDATE]: Term 1 election: Requested vote from peers 8841b443b7e7404582921bf73d4ce2e2 (127.16.21.61:39947), 7036e4af9c2e420f876cacbb8eccae8b (127.16.21.60:45491)
I20260806 10:53:03.529361 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7036e4af9c2e420f876cacbb8eccae8b"
I20260806 10:53:03.529795 7270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:03.533006 7270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9eb519c85446ae8ec75a2c0ba84ade in term 1.
I20260806 10:53:03.533582 7096 leader_election.cc:302] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7036e4af9c2e420f876cacbb8eccae8b, ee9eb519c85446ae8ec75a2c0ba84ade; no voters:
I20260806 10:53:03.534029 7317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:03.545341 7317 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [term 1 LEADER]: Becoming Leader. State: Replica: ee9eb519c85446ae8ec75a2c0ba84ade, State: Running, Role: LEADER
I20260806 10:53:03.545874 7317 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.553833 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9eb519c85446ae8ec75a2c0ba84ade. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.554313 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:03.559692 7331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:03.571452 7220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } attempt: 5
I20260806 10:53:03.575933 7331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 10:53:03.592886 7249 tablet.cc:2369] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:53:03.598023 7098 tablet.cc:2369] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:03.600040 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.564378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.16.21.61:39947)
0806 10:53:03.564551 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.564558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.571068 (+ 6510us) server_negotiation.cc:407] Connection header received
0806 10:53:03.571514 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.571519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.571602 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.571675 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.575902 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.576995 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.586718 (+ 9723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.587428 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.587521 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.592580 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.592587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.592591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.592631 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.596119 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.597215 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.597222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.597225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.597432 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.598125 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.598130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.598134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.598481 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.598716 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.599469 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.599695 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:53:03.609395 7270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [term 1 FOLLOWER]: Refusing update from remote peer ee9eb519c85446ae8ec75a2c0ba84ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:03.588570 7076 master_runner.cc:186] Error getting master registration for 127.16.21.61:39947: OK, instance_id { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" instance_seqno: 1786013582850874 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:53:03.610953 7324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:03.610978 7076 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:53:03.621683 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.569385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.16.21.62:38501)
0806 10:53:03.569584 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.569589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.601211 (+ 31622us) server_negotiation.cc:407] Connection header received
0806 10:53:03.601611 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.601617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.601695 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.601787 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.604457 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.605636 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.608385 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.609039 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.609135 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.614648 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.614655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.614660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.614697 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.618176 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.619102 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.619110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.619114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.619254 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.619682 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.619687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.619690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.620070 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.620341 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.620932 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.621197 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 10:53:03 dist-test-slave-0b58 krb5kdc[7063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.21.61@KRBTEST.COM for kudu/127.16.21.60@KRBTEST.COM
I20260806 10:53:03.636893 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9eb519c85446ae8ec75a2c0ba84ade. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.637210 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.644757 7342 mvcc.cc:205] Tried to move back new op lower bound from 7315511638389833728 to 7315511638209597440. Current Snapshot: MvccSnapshot[applied={T|T < 7315511638389833728}]
I20260806 10:53:03.647362 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.647624 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.650691 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.570958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.16.21.60:45491)
0806 10:53:03.571106 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.571111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.612569 (+ 41458us) server_negotiation.cc:407] Connection header received
0806 10:53:03.612837 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.612846 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.612928 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.612990 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.618222 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.619537 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.625771 (+ 6234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.626711 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.626828 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.638227 (+ 11399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.638237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.638242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.638278 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.642606 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.644991 (+ 2385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.644996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.644999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.645150 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.647602 (+ 2452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.647612 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.647617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.648021 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.648361 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.649343 (+ 982us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.649917 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":51}
I20260806 10:53:03.654194 7331 catalog_manager.cc:1347] Generated new cluster ID: 86e3401bf5924520a33c96ac1ae603bd
I20260806 10:53:03.654299 7331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:03.654728 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.655027 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:03.658537 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.571978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.21.60:45491 (local address 127.0.0.1:34462)
0806 10:53:03.572221 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:03.572241 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:03.572374 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:03.607941 (+ 35567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:03.607949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:03.608453 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:03.608644 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.608660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.615670 (+ 7010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.615676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.617097 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:03.617113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.630536 (+ 13423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.630543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:03.630760 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.631323 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:03.631344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:03.634181 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:03.656018 (+ 21837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.656024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.656027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.656566 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.656969 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:03.656974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:03.656978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:03.657134 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:03.657684 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:03.657689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:03.657828 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 10:53:03.657935 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:53:03.658186 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 10:53:03.659490 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.572022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.16.21.60:45491)
0806 10:53:03.607618 (+ 35596us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.607624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.607648 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:53:03.607711 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.607718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.607799 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.607903 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.613966 (+ 6063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.615256 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.625951 (+ 10695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.630173 (+ 4222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.631158 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.635316 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:03.635326 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:03.635331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:03.635473 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:03.655856 (+ 20383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.656713 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.656717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.656721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.656860 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:03.657281 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:03.657286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:03.657289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:03.657566 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:03.658835 (+ 1269us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.658946 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.659120 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":35447,"thread_start_us":36268,"threads_started":1}
I20260806 10:53:03.673631 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: Bootstrap starting.
I20260806 10:53:03.677443 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:03.678725 7220 log.cc:824] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:03.683689 7331 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:03.685456 7331 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:03.701033 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: No bootstrap required, opened a new log
I20260806 10:53:03.706362 7331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade: Generated new TSK 0
I20260806 10:53:03.711602 7331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:53:03.713037 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.713501 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8841b443b7e7404582921bf73d4ce2e2, State: Initialized, Role: FOLLOWER
I20260806 10:53:03.714118 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } }
I20260806 10:53:03.715452 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.716235 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.716591 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:03.716933 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:03.720379 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8841b443b7e7404582921bf73d4ce2e2"
I20260806 10:53:03.729126 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [term 1 FOLLOWER]: Refusing update from remote peer ee9eb519c85446ae8ec75a2c0ba84ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:03.734222 7195 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9eb519c85446ae8ec75a2c0ba84ade in term 1.
I20260806 10:53:03.734151 7324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:03.801239 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.801724 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.801234 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9eb519c85446ae8ec75a2c0ba84ade. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:03.802248 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:53:03.853734 7367 catalog_manager.cc:1259] Loaded cluster ID: 86e3401bf5924520a33c96ac1ae603bd
I20260806 10:53:03.853850 7367 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: loading cluster ID for follower catalog manager: success
I20260806 10:53:03.864722 7367 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: acquiring CA information for follower catalog manager: success
I20260806 10:53:03.867255 7367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8841b443b7e7404582921bf73d4ce2e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 10:53:03.873378 16468 mini_kdc.cc:348] Time spent kdestroy: real 0.134s user 0.000s sys 0.038s
W20260806 10:53:03.919981 7225 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:03.905738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.16.21.61:39947)
0806 10:53:03.905962 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.905967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.917871 (+ 11904us) server_negotiation.cc:407] Connection header received
0806 10:53:03.918271 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.918277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.918367 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.918439 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.919754 (+ 1315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54628: BlockingRecv error: recv got EOF from 127.0.0.1:54628 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 10:53:03.922654 7300 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:03.914081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.16.21.62:38501)
0806 10:53:03.914291 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.914297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.921011 (+ 6714us) server_negotiation.cc:407] Connection header received
0806 10:53:03.921313 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.921319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.921426 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.921504 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.922429 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57326: BlockingRecv error: recv got EOF from 127.0.0.1:57326 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 10:53:03.929908 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:03.927530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.16.21.60:45491)
0806 10:53:03.927697 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.927704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.928161 (+ 457us) server_negotiation.cc:407] Connection header received
0806 10:53:03.928338 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.928344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.928430 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.928529 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:03.929690 (+ 1161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34472: BlockingRecv error: recv got EOF from 127.0.0.1:34472 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:53:03.977396 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:03.960830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.16.21.61:39947)
0806 10:53:03.961020 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:03.961026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:03.969697 (+ 8671us) server_negotiation.cc:407] Connection header received
0806 10:53:03.970156 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:03.970163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:03.970375 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:03.970498 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:03.971126 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.972249 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.974364 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:03.975056 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:03.975187 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:03.975885 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:03.976539 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:03.976645 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:03.976898 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 10:53:03.977009 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 10:53:04.018563 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.009825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.16.21.60:45491)
0806 10:53:04.010012 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.010019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.016815 (+ 6796us) server_negotiation.cc:407] Connection header received
0806 10:53:04.017074 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.017080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.017231 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.017332 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.018279 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34486: BlockingRecv error: recv got EOF from 127.0.0.1:34486 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:53:04.035135 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.027107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.16.21.61:39947)
0806 10:53:04.027319 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.027326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.028718 (+ 1392us) server_negotiation.cc:407] Connection header received
0806 10:53:04.029003 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.029009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.029098 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.029206 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.029710 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.030935 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.032969 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.033666 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.033797 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.034296 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.034463 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.034567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.034727 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.034813 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 10:53:04.052397 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.050103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.16.21.60:45491)
0806 10:53:04.050296 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.050302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.050814 (+ 512us) server_negotiation.cc:407] Connection header received
0806 10:53:04.050983 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.050989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.051091 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.051203 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.052126 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34496: BlockingRecv error: recv got EOF from 127.0.0.1:34496 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:53:04.064652 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.057814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.16.21.61:39947)
0806 10:53:04.057979 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.057984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.058462 (+ 478us) server_negotiation.cc:407] Connection header received
0806 10:53:04.058630 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.058642 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.058723 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.058817 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.059292 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.060470 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.062375 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.063054 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.063184 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.063684 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.063841 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.063910 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.064184 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.064316 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 10:53:04.076004 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.073852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.16.21.60:45491)
0806 10:53:04.074047 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.074052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.074577 (+ 525us) server_negotiation.cc:407] Connection header received
0806 10:53:04.074727 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.074733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.074837 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.074939 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.075793 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34510: BlockingRecv error: recv got EOF from 127.0.0.1:34510 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:53:04.089692 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.083034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.16.21.61:39947)
0806 10:53:04.083206 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.083212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.083687 (+ 475us) server_negotiation.cc:407] Connection header received
0806 10:53:04.083964 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.083970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.084084 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.084173 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.084641 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.085713 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.087692 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.088401 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.088513 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.088981 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.089102 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.089187 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.089342 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.089413 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 10:53:04.100417 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.098033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.16.21.60:45491)
0806 10:53:04.098267 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.098272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.098765 (+ 493us) server_negotiation.cc:407] Connection header received
0806 10:53:04.098903 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.098908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.099001 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.099093 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.100160 (+ 1067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34522: BlockingRecv error: recv got EOF from 127.0.0.1:34522 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260806 10:53:04.117893 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.110958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.16.21.61:39947)
0806 10:53:04.111115 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.111120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.111632 (+ 512us) server_negotiation.cc:407] Connection header received
0806 10:53:04.111821 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.111828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.111939 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.112021 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.112515 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.113746 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.115665 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.116439 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.116574 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.117071 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.117195 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.117288 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.117476 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.117546 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 10:53:04.126473 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.124184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.16.21.60:45491)
0806 10:53:04.124372 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.124378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.125046 (+ 668us) server_negotiation.cc:407] Connection header received
0806 10:53:04.125244 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.125249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.125329 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.125417 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.126271 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34524: BlockingRecv error: recv got EOF from 127.0.0.1:34524 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:53:04.151564 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.145838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.16.21.61:39947)
0806 10:53:04.146029 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.146037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.146364 (+ 327us) server_negotiation.cc:407] Connection header received
0806 10:53:04.146552 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.146555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.146613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.146712 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.147097 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.147853 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.149841 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.150205 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.150310 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.150872 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.150993 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.151056 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.151224 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.151319 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 10:53:04.158478 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.156621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.16.21.60:45491)
0806 10:53:04.156838 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.156844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.157163 (+ 319us) server_negotiation.cc:407] Connection header received
0806 10:53:04.157335 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.157339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.157406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.157480 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.158274 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34530: BlockingRecv error: recv got EOF from 127.0.0.1:34530 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:53:04.201006 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.194689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.16.21.61:39947)
0806 10:53:04.194924 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.194930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.195566 (+ 636us) server_negotiation.cc:407] Connection header received
0806 10:53:04.195777 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.195782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.195847 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.195924 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.196450 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.197204 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.199213 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.199682 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.199775 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.200239 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.200394 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.200460 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.200620 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.200681 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 10:53:04.209535 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.207381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.16.21.60:45491)
0806 10:53:04.207601 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.207606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.207992 (+ 386us) server_negotiation.cc:407] Connection header received
0806 10:53:04.208175 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.208179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.208248 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.208375 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.209278 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34538: BlockingRecv error: recv got EOF from 127.0.0.1:34538 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:53:04.283417 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.277381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.16.21.61:39947)
0806 10:53:04.277594 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.277600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.277941 (+ 341us) server_negotiation.cc:407] Connection header received
0806 10:53:04.278139 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.278142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.278210 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.278297 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.278792 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.279575 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.281662 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.282104 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.282259 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.282661 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.282776 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.282889 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.283045 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.283133 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 10:53:04.290930 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.289096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.16.21.60:45491)
0806 10:53:04.289287 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.289295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.289667 (+ 372us) server_negotiation.cc:407] Connection header received
0806 10:53:04.289822 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.289827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.289930 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.290042 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.290732 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34544: BlockingRecv error: recv got EOF from 127.0.0.1:34544 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:53:04.291863 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9eb519c85446ae8ec75a2c0ba84ade" member_type: VOTER last_known_addr { host: "127.16.21.62" port: 38501 } } peers { permanent_uuid: "8841b443b7e7404582921bf73d4ce2e2" member_type: VOTER last_known_addr { host: "127.16.21.61" port: 39947 } } peers { permanent_uuid: "7036e4af9c2e420f876cacbb8eccae8b" member_type: VOTER last_known_addr { host: "127.16.21.60" port: 45491 } } }
I20260806 10:53:04.292146 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9eb519c85446ae8ec75a2c0ba84ade [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:04.426761 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.421975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.16.21.61:39947)
0806 10:53:04.422162 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.422166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.422511 (+ 345us) server_negotiation.cc:407] Connection header received
0806 10:53:04.422660 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.422664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.422727 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.422811 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.423257 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.423921 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.425311 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.425760 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.425848 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.426171 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.426292 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.426357 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.426482 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.426542 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 10:53:04.433600 7339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:04.431855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.16.21.60:45491)
0806 10:53:04.432072 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.432078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.432385 (+ 307us) server_negotiation.cc:407] Connection header received
0806 10:53:04.432511 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.432515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.432594 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.432672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:04.433379 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34554: BlockingRecv error: recv got EOF from 127.0.0.1:34554 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:53:04.447827 7314 catalog_manager.cc:1259] Loaded cluster ID: 86e3401bf5924520a33c96ac1ae603bd
I20260806 10:53:04.447901 7314 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: loading cluster ID for follower catalog manager: success
I20260806 10:53:04.450233 7314 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: acquiring CA information for follower catalog manager: success
I20260806 10:53:04.451890 7314 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7036e4af9c2e420f876cacbb8eccae8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:53:04.698050 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.693018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.16.21.61:39947)
0806 10:53:04.693238 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.693244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.693482 (+ 238us) server_negotiation.cc:407] Connection header received
0806 10:53:04.693726 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.693730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.693797 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.693888 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.694348 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.695088 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.696445 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.696854 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.696988 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.697379 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.697475 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.697566 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.697718 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.697796 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:53:04.708608 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.703243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.16.21.60:45491)
0806 10:53:04.703434 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.703441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.703626 (+ 185us) server_negotiation.cc:407] Connection header received
0806 10:53:04.703818 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.703822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.703988 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.704131 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.704578 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.705333 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.706550 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.706949 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.707078 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.707545 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.708058 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.708231 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.708307 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.708384 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:53:04.720920 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.713924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.16.21.61:39947)
0806 10:53:04.714098 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.714103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.714330 (+ 227us) server_negotiation.cc:407] Connection header received
0806 10:53:04.714540 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.714547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.714651 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.714737 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.715575 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.716463 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.718658 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.719276 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.719390 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.720064 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.720214 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.720297 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.720547 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.720623 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 10:53:04.722369 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.713587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.16.21.62:38501)
0806 10:53:04.714906 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.714913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.715482 (+ 569us) server_negotiation.cc:407] Connection header received
0806 10:53:04.715594 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.715600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.715822 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.715980 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.716578 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.717719 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.719827 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.720333 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.720473 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.721153 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.721583 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.721687 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.721964 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.722058 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":1053,"threads_started":1}
I20260806 10:53:04.722383 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:04.715296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.16.21.60:45491)
0806 10:53:04.715475 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:04.715483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:04.715997 (+ 514us) server_negotiation.cc:407] Connection header received
0806 10:53:04.716288 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:04.716295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:04.716378 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:04.716440 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:04.717361 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.718397 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.720436 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:04.721025 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:04.721145 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:04.721670 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:04.721791 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:04.721835 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:04.722082 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 10:53:04.722195 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:53:04.727576 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 7076
I20260806 10:53:04.728513 7139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:04.835258 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 7076
I20260806 10:53:04.859485 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 7151
I20260806 10:53:04.860673 7215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:04.966045 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 7151
I20260806 10:53:04.990165 16468 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu pid 7228
I20260806 10:53:04.991199 7290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:05.098883 16468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLWW7rW/build/asan/bin/kudu with pid 7228
2026-08-06T10:53:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4586 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9474 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (363139 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST